Elasticsearch 1.4.0: Marvel Sense fails with “Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource”

After upgrading an Elasticsearch cluster from v1.3.2 to v1.4.0, using Marvel Sense to run queries against the server would fail with:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://www.server.com:9200/?_=1417024257696.
This can be fixed by moving the resource to the same domain or enabling CORS.

This is because ES 1.4.0 now disables CORS by default in order to patch a security vulnerability. To enable CORS, simply add the following to your elasticsearch.yml file:

    enabled: true

You can lock it down further by setting specific origins (such as localhost):

    allow-origin: /https?:\/\/localhost(:[0-9]+)?/

There are plenty of other options for CORS in Elasticsearch, you can read about them at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html#_settings_2

Spotlight hung on OS X Yosemite with 100% CPU usage

I recently upgraded from OS X Mavericks to Yosemite, and all went well except for one thing: The mds processes were constantly taking 100% CPU, resulting in decreased system performance and decreased battery life. Now I’ve been through this dance before, knowing full well that every OS X upgrade results in a Spotlight re-index to upgrade the Spotlight DB and pick up any new files. But in the past this re-index had taken at most an hour or two on my roughly 80% full 750GB SSD drive. But this time was different — Spotlight was running not for hours, but for days. I decided to let Spotlight run thinking that maybe things were different with Yosemite. After all, Spotlight was one of the big new changes highlighted by Apple in their Yosemite reveal. But after 4 days of 24×7 indexing, I decided that enough was enough and started to investigate.

There are a million different things that can go wrong with Spotlight, and plenty of articles out there on how to fix them. Some of the things I tried were:

  • Verify Disk & Verify Disk Permissions in Disk Utility
  • Fiddle with the mdutil command to disable/re-enable indexing
  • Manually removing the /.Spotlight-V100 directory so that it would be re-created
  • Adding / to Spotlight’s Privacy tab and then removing it to force a re-index.

I tried them all, and none worked.

The symptoms were simple: the mds process was constantly taking 100% CPU. I had a sneaking suspicion that it was hitting a file that it didn’t know what to do with and would just get stuck and not proceed. I was right. Using lsof I checked what files the mds process had open:

sudo lsof -c '/mds$/'

Amongst all the various /System/Library, /.Spotlight-V100, and /Volumes files and directories, I noticed a file that stuck out like a sore thumb:

/Users/MyName/Documents/Subfolder/SomeApp.app/Contents/MacOS/Flash Player

It was actually listed twice in the lsof output. Weird, lets go check it out in Finder. Sure enough, trying to navigate there in Finder caused Finder itself to freeze up at 100% CPU usage. So I restarted Finder with Option-RightClick-Relaunch from the Dock and manually removed the file from Terminal. I then re-started Spotlight and told it to re-index from scratch:

sudo rm -rf /Users/MyName/Documents/Subfolder/SomeApp.app
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
sudo mdutil -i on -E /

This time I could see progress in the Spotlight window’s progress bar, and see files in the Spotlight DB on disk being changed and the total size increasing. About 45 minutes later Spotlight had indexed all 580GB of data. FIXED!

I have no idea why both Spotlight and Finder has trouble with this file. It was never an issue on previous versions of OS X. It was an old custom app built by a friend of mine in 2010. And while mds was stuck on the Flash Player within it, Finder crashed when navigating to directory containing the app itself. Weird.

Some helpful commands for when dealing with Spotlight:

Stop Spotlight:
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

Start Spotlight:
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

Get the size of the Spotlight Database:
sudo du -m /.Spotlight-V100

Get a list of files opened by the MDS process:
lsof -c '/mds$/'


Turns out this troublesome file made its way onto my Time Machine backup as well, causing the *exact same issue* in Time Machine. Backups would start and get stuck on this file, causing the backupd process take up 100% CPU forever until I manually killed the backup. So if your Time Machine is exhibiting similar behaviors as described above, find out what file(s) it is barfing on with:

sudo lsof -c backupd
sudo lsof -p [PID]

Then use Time Machine’s “Delete All Backups Of…” functionality to remove it. Kick off another run of Time Machine and it should make it all the way through.

If an Apple Engineer comes across this blogpost and would like a copy of the troublesome file, hit me up.