this rootless Python script rips Windows Recall’s screenshots and SQLite database of OCRed text and allows you to search them.
Hilarious to me that it OCRs the text. The text is generated by the computer. It’s almost like when Lt. Cmdr. Data wants to get information from the computer database, so he tells the computer to display it and just keeps increasing the speed — there are way more efficient means of getting information from A to B than displaying it, imaging it, and running it though image processing!
I totally get that this is what makes sense, and it’s independent of the method/library used for generating text, but still…the computer “knows” what it’s displaying (except for images of text), and yet it has to screenshot and read it back.
To be fair, Data was designed to be like a human, and was made in the image of his creator. He has a number of design decisions that are essentially down to his creator wanting to create something like a human. Including that which you describe.
Data was never intended to work like a PC, it’s very normal that he can’t just wirelessly interface with stuff.
It happens the same on android for some reason
Like 5-8 years ago the google assistant app was able to select and copy text from any app when invoked, I think it was called “now on tap”. Then because they’re google and they’re contractually obligated to remove features after some time, they removed this from the google app and integrated it in the pixel app switcher (and who cares if 99% of android users aren’t using a pixel, they say). The new implementation sucks, as it does ocr instead of just accessing the raw text…
It only works fine with us English and not with other languages. But maybe it’s ok as it seems that google’s development style is us-centric
Now on Tap also used OCR. Both Google Lens and Now on Tap get the same bullshit results on any languages that are not Latin. Literally, Ж gets read as >|< by both exactly the same.
They changed it, in the beginning it was using the text and not ocr
For example this app could be set as assistant and get the raw text https://play.google.com/store/apps/details?id=com.weberdo.apps.copy
But only the app set on system as assistant can do it
I was very disappointed when they changed it around 2018 as it produced garbage in my language when it was working so good…
Having worked on a product that actually did this, it’s not as easy as it seems. There are many ways of drawing text on the screen.
GDI is the most common, which is part of the windows API. But some applications do their own rendering (including browsers).
Another difficulty, even if you could tap into every draw call, you would also need a way to determine what is visible on the screen and what is covered by something else.
Hey, yeah… why aren’t they just tapping the font rendering DLL?
…are they tapping the front rendering dll??
Iirc chrome stores your local cookies/session in a place malware could also attack. Probably the same idea for other browsers.
I’m not sure I fully understand the issue here. If we’re ok with that info being trivially retrievable by a bad actor, why isn’t this ok?
Like I get you may not like it, and it’s a target, but there are already lots of targets that have gotten a pass based on user permissions. Is it just the breadth of potential info? With the cookies you could potentially log into someone’s bank account.
First, false equivalency.
Second, we’re not okay with cookies and session being in a place that could leak — it’s why we’re doing everything possible to stop that from happening (I mean GDPR alone is one effect of this).
Third, the fact that you can’t see a difference between cookies, which actually can be secured via proper encryption and signing, and a literally unencrypted database driven by OCRed screenshots (taken every couple of minutes) that requires an opt-out and is a very small slippery slope to that data making its way back to Microsoft’s own servers for their own greedy pursuits….then I’m not sure what to tell you.
Recall is wrong. And it’s indefensible. Period.
If you think it’s okay, then feel free to open everything up to Microsoft of who you are and what you do on your Copilot+ PC. I, for one, among many, will choose to secure my information as best as possible, including never using another Microsoft product again, if at all possible. And I’ve already done so for myself.
GDPR has little to do with this. People use site cookies to remember sessions and not have to login again, etc. I’d guess most browser users use and want to use this functionality. If you’re fully opting out to not even have persistent sessions, I’m guessing you’re in the far minority of users here.
I’m not aware of any non-trivial readily available built-in encryption for cookies. There are easy to find libraries that exist to just pull out cookies (stored locally including session tokens).
To clear up a bit more misinformation from your response: this is an offline feature. The data doesn’t go back to Microsoft. It works even if your computer is disconnected from the internet. If you consider their word to be a lie on this part, that’s you’re right to believe, but until proven, isn’t a fact.
GDPR has little to do with this
Not at all true, GDPR is the exact reason why you see all of the sites these days letting users know that their site stores cookies and requesting acceptance of it. Hence why I said we, as a global society, are trying to do something about this, even if it’s something as simple as cookie use disclosure on sites – it’s a start.
If you’re fully opting out to not even have persistent sessions, I’m guessing you’re in the far minority of users here.
Never once said I did.
I’m not aware of any non-trivial readily available built-in encryption for cookies.
You’re correct, data-at-rest encryption doesn’t exist for cookies, but data-in-flight does with SSL. Also, signing cookies and samesite origin is a thing being done these days, which makes them quite improbable, if implemented properly, to be hacked for any actual use in terms of leaking logins to said sites.
this is an offline feature. The data doesn’t go back to Microsoft
For the moment, that’s what they say, yes. And that’s the problem, especially since it’s turned on, by default. This – is not – something – Microsoft has earned trust for.
But you are free to believe them all you want – the rest of us who have seen what Microsoft has done these past 40 years use that as a guide to judge – and history is usually a very good judge.
browser data is a potential liability, sure, but you have tools to manage it. you can delete pages or entire websites, you can use private windows, you can purge history older than 6 months or something like that, and at least a few browsers have a “forget” button that wipes out the last two hours of history. similar deals with cookies and other data, and we’ve collectively decided the benefit of having browser data is worth the risk.
not so here. Recall is a record of everything you’ve ever done on your PC. you can’t selectively delete things like you can with browser history, the app and website exclusion is only as good as whatever Recall is using to detect apps and websites, and you can’t redact sensitive info after the fact. people are generally okay with browser history and data because they know they have fine-grained controls to manage it, controls Recall doesn’t have
So if they had a ui with buttons to ‘pause for X length (could be forever)’, buttons to 'forget last X length (once again could be forever), but everything else stayed the same, would it be acceptable?
Like I’m genuinely curious here.
When you go on the internet you are accessing content on other people’s computers. You are saying, “I want such and such document”. There’s an inherent lack of privacy in browsing the internet. You can try to be private about it, but ultimately you’re not changing that you’re requesting data from other people’s computers and sending them data.
When you are doing something else on your PC besides browsing the web, Recall is still taking screenshots and tracking you. What apps you use, pictures you view, and many other things that might be completely offline and you don’t necessarily want a history of stored on your PC, with screenshots and searchable summaries. Do you want each and every one of your fap sessions recorded? Why would you want any of your offline activity recorded?
What if you forget to pause this feature and someone finds these screenshots? Who cares, right? What if your a closeted gay teen living in a conservative country and your family finds the history?
Then there are people who don’t understand computers using offline business software for accounting, or whatever, and even if they store their data files on an encrypted drive or something, Recall is taking screenshots of everything they do. If they don’t even know its happening, their PC could have years of data that could be stollen from them at any point in the future. Even if they never open those encrypted files again. Obviously, if their computer is pwned, then the hackers could just take the enencrypted files when they’re next accessed, but Recall snapshots everything all the time, even if you delete it.
Edit a self nude photo on your PC and forget to turn off Recall, and then layer decide to delete the photo… Too bad, Recall still has it.
It’s a feature that’s… ok if you want it, but it should not be part of the operating system, and it definitely shouldn’t be opt-out. It should be an app that you install with deliberate purpose if and only if you want itand understand the security and privacy risks.
Microsoft instead wants to install it by default and probably turn it on by default. Even if it ends up being opt-in, MS has a long history of asking people to enable features in misleading ways. And the vast majority of Windows users don’t understand computers!
if i were designing a recall program, here’s how i would do it: it would take a screenshot every five seconds, OCR it, then run it through local quantized image recognition and word association neural networks, and then toss everything into a CryFS vault. when launching the recall program, you have to provide the password to unlock the vault so it can read and write to it. it can only run in the foreground (so you have to keep the window open for it to run, no closing it and forgetting about it) and it will display a status indicator in your system tray that provides a menu to pause or stop recording. afterwards, you can mark any text or region of the screen for redaction, and it’ll redact it across all screenshots and delete it from the database; you can delete individual screenshots or entire periods of time; and there will be an easily accessible self-destruct option that shreds the database (i.e. overwriting it with random garbage 21 times before deleting it off the disk). this is all offline and the application will not request network access
i’m just making this up on the fly, so there are absolutely security and privacy considerations I absolutely forgot about, but this is the bare minimum i would like to see
What an unexpected turn of events.
In a hilarious and infuriating side note, MS is obviously doing their absolute best to blame-shift here.
It’s code. It’s a project someone made to graphically illustrate and demonstrate, in the wild, why the entire concept of MS Recall is an absolutely awful, foundationally-flawed idea. It is not a “hacker tool”. The MS c-suite and board members are just pissed that stock go down as a result of their stupidity, and they’re looking for people to blame who aren’t themselves.
Where is the blame shifting? The article says they made no comment and the only MS quotes are just random pr feature blurbs
Dude the headline:
this hacker tool
It’s absolutely not a “hacker tool”. It’s a proof of concept. It’s just code. The author and/or editor is leaning on ingrained negative kneejerk reactions from less knowledgeable members of the general public towards the term “hacker”.
So that’s not Microsoft, that’s Wired doing that. Also it IS a hacker tool. It’s a tool to automate the scraping of data and sending it somewhere.
He’s a white hat hacker, releasing the tool to raise awareness. If he was a black hat hacker he’d be holding onto it and praying Microsoft goes through with release so he could use it to compromise systems.
I don’t see any blame shifting at all
