Google

2015-11-20

PowerShell Function to Fix System Path

Before Windows 7, max length of the PATH environment variable was 1024 chars. Many years ago, we found ourselves troubleshooting a huge problem in our environment that was caused by faulty install scripts that were appending the PATH variable many times taking the path length over the limit and causing all sorts of problems as PATH was cut at 1024.

Windows 7 increased the length but I've seen the same problem in other environments. I have seen a PowerShell question over at StackOverflow today, and apparently the person was running into the problem because the max path length was exceeded. When that happens, Windows just ignores the characters after the max length is reached.

I had written a PowerShell function that would search for and remove the duplicates that would increase the length of PATH unnecessarily. I put it up as a gist at github & sharing it here. Hopefully it helps someone.

2015-11-15

ConvertFrom-String or Not

One of the new cmdlets in PowerShell v5 is ConvertFrom-String. It's using a Microsoft Research Technology called "FlashExtract", a machine learning technology. I was hoping that it would finally rescue us from RegEx! ~/Yay/Heh/g

I've seen several MVPs write about it, mostly before Microsoft released the final version but some recent as well:
The premise is that, you provide samples of the data and specify what you want to extract out of it, and FlashExtract engine is supposed to be smart enough to evaluate the template you provided and get the properties you want. If only life was that simple...

Alright here is our victim... Let's take a look at our WinSxS (component) folder. People always complain about how big it is and how unwieldy it gets over time as more and more patches come down. It's a tempting target to clean up old patches and other binaries but also the shortest path to breaking your Windows if you remove a needed file.

Anyway, I am more interested in creating an easy to read PowerShell object based database out of it.

They look like this:
C:\windows\WinSxS\amd64_aspnet_compiler_b03f5f7f11d50a3a_10.0.10586.0_none_8037b2faf04e42be

Microsoft tells us how it is formatted - What's that awful directory name under Windows\WinSxS. Oh, it's supposed to be a 'friendly-name' with this format:

proc-arch_name_public-key-token_version_culture_hash

So, the name is separated by a bunch of underscore characters. This makes our life so easy to parse it, right? Wrong! "name" too can have "_", as well as a bunch of other chars like "." and "-".

Still, after two minutes of RegEx tinkering I got that function and it worked perfectly:

PS C:\Users\Adil> (Get-WinSxSRegex)[1,100,1000]

Processor      : amd64
Name           : 1394.inf
PublicKeyToken : 31bf3856ad364e35
Version        : 10.0.10586.0
Language       : none
Hash           : 87b4eef7b03f2543


Processor      : amd64
Name           : c_bluetooth.inf.resources
PublicKeyToken : 31bf3856ad364e35
Version        : 10.0.10586.0
Language       : en-us
Hash           : 14d0dc285c89e9ba


Processor      : amd64
Name           : microsoft-windows-b..iagnostic.resources
PublicKeyToken : 31bf3856ad364e35
Version        : 10.0.10586.0
Language       : hu-hu
Hash           : 2d6242e40cbc20a9

The example above is showing the 1st, 100th and 1000th records. Luckily, they are good examples to demonstrate the challenge that ConvertFrom-String cmdlet faces.

So, I took a couple of folder names and started to put the property names as I would like them.

{Processor*:amd64}
This is supposed to collect the Processor, but if I do not use "*",  it does not work. It seems that when the first matching line is found, that becomes a record, and to be able to get more records, we need that * in there.

{Name:1394.inf}
This will work, but only for some. See, because it starts with a number, without any other sample for Name, engine thinks we want something that starts with 'numbers', and gives us only matching folder names that has numbers in the 'name' section. so, we need to give different Name samples.

{PublicKeyToken:31bf3856ad364e35}
To a human eye, this is pretty easy to locate. It's a hex, and if we were using regex, we would use something like '[a-fA-F0-9]+', in other words one more numbers or characters that stops are between 'a' and 'f'. Well, we are not dealing with a human here.

{Version:10.0.10586.0}
OK, this must be the easiest to identify however you look at it. 4 digits with "." between them

{Language:none}
Language is a bit tricky. It could be none or some international language designation like "en-us" for US English.

{Hash}
Finally, we have the hash, which is pretty much like the PublicKeyToken

I will leave it here. I tried a couple of alternatives in templates, and providing more was not getting me anywhere as I kept on getting error messages telling me ConvertFrom-String cmdlet could not come up with a way to extract data using my template and suggested I email Microsoft. I did, though I am not holding my breath on that.

I really like the concept of ConvertFrom-String, and in my opinion, It would be one of the most powerful features of PowerShell but to me, it feels like it is just not there YET.

Sample templates I used are below. If you will use the functions, pipe them to |out-gridview. It is pretty easy to detect problems that way.


2015-11-01

Clear Recent Documents List on OS X

TL;DR: 
Remove the relevant .sfl file located under the following directory:
~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/

- * -

It's Halloween! Time for scary stuff. As the tech admin of the house, my job tonight was to make sure a series of old scary movie classics, think of Poltergeist , can be streamed to TV.

I decided to clear the existing list of QuickTime Player's Recently Opened Files and pre-populate the list with the scary movies. This can be done from GUI:

QuickTime Player > File > Open Recent > Clear Menu

Nothing is wrong with that, but I wondered how I would do that from Command Line. Interestingly, I could not locate a setting in a .plist. If you Google, you will find suggestions similar to below, but on El Capitan (10.11), these settings did not exist


adil-imac-1:~ adil$ defaults read com.apple.QuickTimePlayerX.LSSharedFileList RecentDocuments
2015-10-31 23:35:21.932 defaults[12222:1913397] 

The domain/default pair of (/Users/adil/Library/Preferences/com.apple.QuickTimePlayerX.LSSharedFileList, RecentDocuments) does not exist

So, to find which file I needed to look at, I decided to use good old, 'find' command. The technique is well known:
  • First create a temporary file
  • Then take action (launch a movie) to cause a file change, as new movie is added to the recent items
  • Lastly do a find for all files newer than the temporary file we created


touch /private/var/tmp/now
sudo find / -newer /private/var/tmp/now

This, usually works, but file list could be larger. In my test, it returned 120 files in that short time frame.

There are other issues with this approach:
  • It's relatively slow!
  • If you have external drives mounted, you might need to try -x to avoid scanning them

There is a better way! Try Spotlight from command line: mdfind.  

Approach is pretty similar to find, except there is no need to create a temporary file. 
  • Make the change
  • Immediately run mdfind telling it to look for changes in the last, say 15secs.

mdfind 'kMDItemFSContentChangeDate>$time.now(-15)'

k: kind
MD: Metadata
FS: File System
$Time variable is explained in the Apple Developer Documentation link below

Apple Developer documentation on MetaData Query Expression Syntax has some examples on using MetaData information. There are a few examples here as well.  

mdfind command pointed to this file:

~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.apple.quicktimeplayerx.sfl

What's .sfl? Not exactly sure but it probably stands for 'Shared File List' and apparently is meant to hold "Bookmark Data".

The content of the .sfl file is kinda cryptic. Incidentally, I found a blog post from a couple of days ago that goes about Exposing "BookmarkData"

When you manually clear the Recent Documents, the .sfl file becomes shorter in size. However, as there does not seem to be a tool to manipulate the file yet, deleting the file seems like a safe way to clear bookmarks.

Just in case people are wondering, yes, VLC has 'org.videolan.vlc.sfl' located at the same folder, and removing that file cleared the VLC recent docs as well. 


2015-09-13

No Birthday Reminders for People You Don't Know

I made it a habit to add my Google account to "Internet Accounts" in OS X, which also enables me to see calendar events on the notification window when I am on my Mac. Problem is that I get bombarded with birthday reminders for people I do not personally know.




My first shorthand solution was to uncheck "Calendar" from Google Account in the OS X Calendar application.


That solves the problem on Mac, but I am 'still' a Chrome user and see these alerts on Chrome too.

There is a long article on Techcrunch from early 2015 complaining about this issue, but a note at the end that Google was working on a solution. No update after that, but I think Google has fixed it now, and you can stop these alerts at the source :


  • Go to google.com/calendar, you should see Birthdays under Calendars
  • Under My calendars, click on dropdown box to the right of Birthdays and select "Calendar Settings" or "Edit Notifications"


  • Select "Contacts only" option next to "Show birthdays from"


  • Hit "Save" at the bottom


Enjoy!

2015-04-25

Chrome Remote Desktop Eating Up Free Space

Last night, I noticed my iMac was a bit unhappy as it had run out of (250GB) SSD space. I recalled that the night before I had installed Google Drive and left it synchronizing and also had not yet moved the new Photos app to larger secondary drive, which was importing more and more photos in the last few days. By default Photos app installs at ~/Pictures. So they were both eating up precious SSD space.

To  move the Google Drive to secondary drive:
  • exited the app, 
  • moved the ~/Google Drive/ folder to second drive, 
  • restarted the app
  • Disconnected and reconnected the account choosing the new location where I moved the directory

To move the new Photos App to the secondary drive:
  • exited the app
  • moved the ~/Pictures/Photos.Library.photoslibrary to secondary drive, next to my iPhoto Library
  • Restarted the app
  • It detected the new location and prompted me to select the library.



April 22 was the earth day. So, today, I took my daughter to her school and together with her friends and their parents we cleaned up the school playground. 

I came back home and checked the Disk Utility to see if my clean up from yesterday had tidied things up enough. To my surprise, there was still a ton of space consumed by "something".

I launched my favorite tool "Disk Inventory X" which quickly detected that I had a 150Gig log file: 
/var/log/org.chromium.chromoting.log.0

Looking at the folder, it seemed that the log file was roller over, but the old log file somehow kept going and consumed the whole disk. This file is created by Google's awesome "Chrome Remote Desktop" extension which I really like to use. 

Unfortunately, there is currently an open issue for this case: 452121. It's not clear when this bug was introduced but it has been there for the last few versions of chrome (38..42) and some people have seen it eating up to a TeraByte of space, unless "remoting_me2me_host" process is killed and restarted with: 

killall -HUP remoting_me2me_host

Well, thanks Google for giving a reason to clean up ;)

2015-03-11

Automount NAS share on Mac

NAS

It's been several years since I was sold on using a NAS to solve the data sharing problems at home. QNAP TS-439 Pro was my choice. Frequent firmware updates kept it up to date and relevant. When something becomes popular, say Google Drive, you get a firmware update that integrates it to NAS. One firmware for all versions of QNAP, new and old. So, your old hardware does not become irrelevant in a year or two.

SMB Versions / Issues

Traditionally AFP (Apple Filing Protocol) was the protocol preferred on Macs for network access. Of course, Windows used SMB (Server Message Block) protocol of Microsoft. Currently Windows 8.1 uses SMB v3. With Maverick (v10.9) release, Apple backed away from their own AFP protocol in support of SMB v2, and that was a problematic release to say the least. With 10.10 Yosemite Mac OS X supports v3 as the default sharing protocol.

QNAP, unfortunately, does not support SMB 3.0. With the release of firmware 4.1.0 Build 0605, QNAP changed samba default protocol to SMB 2.0

SMB Nightmare
After upgrading my Macs to 10.10, I started connecting to my QNAP via SMB 2.0, and problems started. The biggest issue was that whenever I wanted to copy a bunch of files to NAS, it would start copying, and after the first one or two, I would get an error that read:

"Operation cannot be completed because file is in use",

and copy would fail.

I could workaround the problem by going to terminal and use 'scp' to transfer files to NAS, but it was not that convenient.

Many people complained about this problem on Apple forums. None of the solutions offered helped in my case, so I went back to AFP, and that problem disappeared. I still see other issues when, for example, editing a word document: Word somehow cannot save the file the first time, then lock is released and file is saved. That's more of an Microsoft office problem, rather than AFP/SMB though.

Automounting NAS

One problem that annoyed me was that even with AFP, I had to connect to my NAS over and over. It was not as bad as SMB which kept on prompting me, but it was still an hassle, especially on my Macbook, which is not always near my wifi to connect to home NAS.

Over the weekend, I took another look at Apple's Autofs: Automatically Mounting Network Files Shares whitepaper  and implemented it for my macs.

The implementation is simple. Apple now has /etc/auto_master file, which can lookup other files for indirect mounting information and automatically carry out the mount instructions.

adil-imac-1:~ adil$ ls /etc/auto*

-rw-r--r--  1 root  wheel   149 Sep 19 03:15 /etc/auto_home
-rw-r--r--  1 root  wheel   233 Mar  8 17:47 /etc/auto_master
-rw-------  1 root  wheel   203 Mar  8 21:32 /etc/auto_qnap
-rw-r--r--  1 root  wheel  1935 Sep 19 03:15 /etc/autofs.conf

Now let's take a look at auto_master, notice the last line I added

adil-imac-1:~ adil$ cat /etc/auto_master
# Automounter master map
#

+auto_master  # Use directory service
/net   -hosts  -nobrowse,hidefromfinder,nosuid
/home   auto_home -nobrowse,hidefromfinder
/Network/Servers -fstab
/-   -static
/-   auto_qnap

And here is what /etc/auto_qnap look like:

adil-imac-1:~ adil$ sudo cat /etc/auto_qnap

/opt/mnt/myshare -fstype=afp afp://MyUser:MyPassword@MyServerIP/MyShare
/opt/mnt/public  -fstype=afp afp://192.168.1.5/Public

My initial thought was that I would use /Volumes to mount the folders, but that's really a folder Mac uses for dynamically mounting everything and auto-unmounts them. It's more like a temp directory. So I decided to use a folder more common in linux distributions /opt/mnt, which did not exist on my system of course, so I created it:

sudo mkdir -p /opt/mnt

then I created the auto_qnap file and changed its permissions to be read/writeable by root only, as it might need to contain password to connect to certain network shares.

sudo touch /etc/auto_qnap
sudo chmod 600 /etc/auto_qnap

The format for AFP and SMB are similar:

/Local/Mount/Folder   -fstype=afp    afp://MyUser:MyPassword@MyServerIP/MyShare
/Local/Mount/Folder   -fstype=smb    ://MyUser:MyPassword@MyServerIP/MyShare


I used IP address as it is static, but you could use FQDN of your Network Server. Once this is all done, you can run the following command to clear the cache and verbosely execute the mount command:

sudo automount -vc

I then created a symlink to the new mount points and put them in the Finder Sidebar:

ln -s ~/Pictures/PicsOnNAS /opt/mnt/public/pictures

Using this setup, I could now tell Flickr Uploader Mac Client to watch ~/Pictures/PicsOnNAS folder, and automatically upload pictures when I add a new ones there.

There you have it. It's not as easy as it is for Windows which automatically detects connection state of the mapped network drives, but close enough. Enjoy!

2015-03-10

Flickr Uploadr for Mac


Last year Yahoo revived Flickr, and they offered something none other are offering even now: 1TB free space for your pics and videos. I had my Flickr account from early 2000s that, like everyone else, I had not used for ages. I started using Flickr again...

Yahoo offered Automatic Camera Upload feature from phones, which is pretty much what everyone else was offering and it had a couple of hick ups especially when new iPhones with newer iOS arrived, but other than that it pretty much worked OK.

I kept on using it in tandem with Google+, photos uploading to both. One nice feature I really liked about Flickr was that it allowed me some family members who live worlds apart to my 'family' and automatically share pictures with them when pictures were uploaded, which is not a something I could do with Google+



This setup worked fine 'going forward' but until now there was not a good way of uploading pictures. A few weeks ago, Yahoo released Flickr Uploadr for Mac (beta) and I found out that it works perfectly for my use case.


It can use iPhoto, which I do not really use. But more importantly, it can upload folders of pictures and it creates an Album with the name of the folder.

The way I organize my pic folders is like this:

YYYY > YYYY.MM > YYYY.MM_Event_Location

So when I upload a year, it creates albums with the name YYYY.MM_Event_Location


Of course not all my folders were perfectly named like that, sometimes I had to use PowerShell to fix them (may be subject of another blog post).

It not only uploads them once, but can upload any new pictures you put in those folders. So, if you have an upload folder, for example, anything you drop in there will be automatically uploaded.

It also has de-duplication. Not a very smart one at the moment, so if you have the same picture with a different size, it will not detect that but if you have the same picture in more than one folder, it will detect that and will not upload it again.

I also had to deal with (SMB) connection issues I was having from my iMac to my QNAP NAS where I keep gigs and gigs of pics, and came up with a method to Automatically mount the NAS whenever I needed it without bugging me with credentials and stuff (definitely another post).


Here is link to feedback forum for the product: https://www.flickr.com/help/forum/en-us/72157649816391975/

That's all for now! Enjoy, and let me know what you think if you try it.


2015-02-22

Fixing Home and End buttons in Mac Terminal

I hate it when Terminal in my mac scrolls up when I hit 'Home' key, or all the way to the bottom, when I hit 'End' key. As most would do, I expect 'Home' key to take me to the beginning of the line, and 'End' to the end of it.

I can of course use 'control+a' and 'control+e' to do that, but it's tough to beat muscle memory.

There is a Stack Exchange 'Ask Different' entry for this, but the keys mentioned there for terminal were not working for me: https://apple.stackexchange.com/questions/16135/remap-home-and-end-to-beginning-and-end-of-line

Terminal Keyboard Binding




What seems to work for me is:
* \033[H -> Home
* \033[F -> End

These can be set from Terminal > Preferences > Keyboard as shown in the screenshot. Enjoy!

2015-01-22

How to move iPhoto Library to a different drive

I wanted to move my iPhoto Library from my SSD disk to secondary SATA disk on my iMac.
It's pretty easy to do:

  • Create a folder in the target drive
  • Hold down Command button and move the "~/Pictures/iPhoto Library.photolibrary" directory to the target drive 
  • Double click on the "iPhoto Library.photolibrary" directory to launch iPhoto. 

That's it! It remembers the directory next time you launch it.

















In the screenshot above, you can see the iPhoto Library directory at the bottom, and the target directory (/Data/pictures) on the secondary (SEAGATE) drive at the back right while library is being moved.

Note: My iPhoto version is 9.6. Also found this KB article from Apple describing the same process, so this seems to work for other versions as well: http://support.apple.com/en-us/HT1229