The Google Pixel 2 and Pixel 2 XL are official after many months of leaks. One of the more interesting (and controversial) features is "Now Playing," which detects music playing in the background and shows you what's playing on the lock screen. We first heard about this feature a few weeks before launch, but we didn't have much information about the feature apart from Google telling us that it can work offline without sending any data to the cloud (the latter is especially important in light of recent revelations regarding the Google Home Mini). After digging into the Now Playing feature, we've discovered that the feature is based on years-old technology called AmbientSense which promises minimal battery drain.
Google Pixel 2's Now Playing Feature
We were first tipped off about this when we analyzed the Pixel Ambient Services application, which is available on the Google Play Store.
Pixel Ambient Services (Free, Google Play) →
But it wasn't the app itself that made us aware of the AmbientSense connection. Instead, it was the name of the APK pre-installed in /system/priv-app on the Google Pixel 2. Called AmbientSense, the APK matches the name of a technology described in a research paper presented at the 2013 IEEE International Conference on Pervasive Computing and Communications Workshops by researchers M. Rossi, S. Feese, O. Amft, N. Braune, S. Martis and G. Tröster.
What is AmbientSense and how does it relate to "Now Playing?"
We found a webpage that displays the first page of this paper here. According to the paper's abstract, AmbientSense is a "real-time ambient sound recognition system on a smartphone." What's most interesting about AmbientSense is that it can be implemented as an Android app and only needs access to the device's microphone to analyze ambient sounds.
There are two processing modes described in the paper: autonomous and server mode. Autonomous processing occurs on the smartphone only by comparing audio samples against a locally stored database. In comparison, server mode sends audio features to a server which then sends classification results back. Clearly, Google's "Now Playing" feature is probably running AmbientSense in "autonomous" mode as it can work offline.
The paper goes on to describe how the team of researchers tested recognition performance, runtime, CPU load, and recognition delay under both autonomous and server mode recognition in a set of 23 ambient sound classes. They found that the AmbientSense app ran for up to 13.75 hours on a Samsung Galaxy SII and up to 12.87 hours on the Google Nexus One. Keep in mind how old these devices are; the Google Nexus One was released in 2010 and is a dinosaur in comparison to the Pixel 2. We can only imagine how much AmbientSense has been refined through Google's testing.
Is it Possible to Port the Now Playing Feature on non-Google Pixel 2 Phones?
I can't make any promises yet, but I think it's possible. We're working with XDA Recognized Contributor Quinny899 to make it happen. In order to get the Now Playing feature working on the first generation Google Pixel/Nexus smartphones, there are a few things that we believe are needed:
- Pixel Ambient Services (AmbientSense.apk)
- Audio Matching database
- Some missing libraries
- SystemUI modifications to ambient display
- Root access (to push the above files to /system)
Screenshots credit: Kieron Quinn (Quinny899)
We currently already have the audio matching database in our possession, called "matcher.leveldb." It's a 53MB storage library based on Google's LevelDB. This is the database that AmbientSense relies on to do audio matching in autonomous mode.
As for the libraries, we know what they're called and where to look for them, but it will take some time before we can get our hands on a Pixel 2 to extract it.
Finally, SystemUI needs to be modified because the "Now Playing" feature writes text to ambient display—something which is currently not possible on the ambient display feature found on the first generation Pixel.
As for getting this working on non-Google phones, we'll test that after we get it working on the Google Pixel and Nexus phones. If or when we make a breakthrough on getting this feature working, the first place you'll know about it is the XDA Portal—so stay tuned for more!
0 Response to "Google Pixel 2’s “Now Playing” Feature uses AmbientSense to Minimize Battery..."
Post a Comment