The ChatGPT client now supports "screen reading programming".
Starting today, Mac Plus users can directly have the ChatGPT client read the code in their development tools.
Currently, this feature supports five programming tools, including the commonly used Xcode and VS Code.
User Mckay Wrigley said it feels like a dream and can't wait to see the future development.
Another user said that the cases promoted by OpenAI are too simple, and this feature is actually more powerful than the demonstration.
At the same time as the Mac version update, the long-awaited desktop client for Windows users has also been fully launched.
Mac Plus users can "screen read programming"
Starting today, the Mac client of ChatGPT can "screen read programming".
In OpenAI's official promotional video, the demonstrator asks to add a tab to an example app.
You can see that ChatGPT automatically imports the existing code into the conversation and generates new complete code according to the requirements.
In fact, this is the "Work with Apps" feature launched in the Mac version of the ChatGPT client, which first covers the results of programming tools.
Specifically, this feature currently supports three IDEs or editors (Xcode, VS Code, and TextEdit), as well as two terminal-type tools (Terminal and iTerm).
Before using it, you need to enable "Accessibility" permission for the ChatGPT client in the system settings.
In the three editors, ChatGPT will process the complete code contained in the foremost window, until the window length limit is exceeded.
If a portion of the content is selected in the editor, ChatGPT will focus on the selected part while processing the entire window.
For the two terminal-type tools, ChatGPT will process the last 200 lines in the window.
Currently, this feature is available to Plus and Team subscription users, and will cover Enterprise and Edu users "in the next few weeks".
From the name "Work with Apps", this feature may not be limited to programming tools, and may support more applications in the future.
Given the rumor that OpenAI plans to launch its intelligent agent next year, "Work with Apps" may be OpenAI's preliminary attempt in this direction.
However, the "Work with Apps" feature is currently implemented by calling the "Accessibility" interface of Apple, rather than reading the screen content visually.
And the new content generated based on the existing code cannot be directly imported into the programming tools, it needs to be manually copied and pasted.
So, the user experience of this ChatGPT feature has received some complaints.
In comparison, OpenAI's competitors Anthropic and Google have already had screen operation functions based on visual perception.
Hopefully, by the time OpenAI's Agent is truly launched, the user experience can also be improved.
The Windows client has been fully launched
In addition to the Mac version feature update, this time OpenAI also brought good news to Windows users--
The Windows client has been fully launched, and free users can also use it.
The fully launched client already supports advanced voice mode, and the newly launched search function is also in place.
Similar to option+space in Mac, on the Windows system, you can call ChatGPT through the Alt+space key combination from any interface.
In the pop-up window, you can enter text, upload screenshots, take photos with the camera, or upload local files.
The Microsoft App Store interface shows that the Windows client was launched on September 7, and according to the information on the OpenAI website, the beta version was opened to Plus, Team, Enterprise and Edu paid users in October.
But in any case, starting today, both paid and free users can experience the Windows client.
In the comments section of OpenAI's official Twitter announcement, many users asked if it supports Windows 10:
Regarding this issue, the Microsoft App Store shows that it is supported, but the sub-version number needs to be 17763.0 (also called the October 2018 update or version 1809) or above.
Reference links:
[1]https://techcrunch.com/2024/11/14/chatgpt-can-now-read-some-of-your-macs-desktop-apps/
[2]https://help.openai.com/en/articles/10119604-work-with-apps-on-macos
[3]https://help.openai.com/en/articles/10003026-windows-app-release-notes
This article is from the WeChat public account "Quantum Position", author: Kelsey, authorized by 36Kr for release.
