Tuesday, January 18, 2011

Garmin FIT activity splitter to eliminate large time gaps

I've gotten useful code on my third project for processing FIT files using Kiyokazu Suto's Garmin::FIT package for Perl. First, I described fit_to_cols, which extracted selected data from FIT files and formatted it in a space-delimited file. Next I described fit_filter_motor_segments which attempted to identify segments where the Garmin was accidently left running in a fast car or train. That project's still being refined, as I described last post.

Here is perhaps the most useful of the three, fit_split_on_gaps, which finds gaps of some specified minimum duration (default 8 hours, or otherwise specified with the -tgap option) and splits the FIT data into multiple sub-files at any gaps found which meet or exceed this threshold.

This and the other codes can be found here, on Google Docs.

It's fairly common in my experience to forget to "reset" my Garmin between activities, despite having set it to warn me at the start of a ride if I have not. GoldenCheetah, for example, contains some nice code to deal with this problem, allowing rides to be split at selected gaps. However, since I've not been using a power meter since last summer, I've been using Strava a lot more than GoldenCheetah. And Strava lacks and ride-splitting capability.

Enter fit_split_on_gaps. Its default behavior is to take each FIT file and, if it finds any gaps between meaningful data (ignoring the "power down" and "power up" records which sometime pollute idle periods), it partitions the original file into sub-files, naming each according to the Garmin convention of "yyyy-mm-dd-hh-mm-ss" with a "_split.fit" suffix (although the suffix can be changed) in the same directory. Alternately the files can be written to a separate directory with or without (default) a suffix.

There's a few options. The most important is "-h" which prints help. I won't try and describe the other options, as the code may still be modified and these may change. However, so far there's two primary modes:
  1. To take one or more input file, and produce output files (by default only if gaps are found) of individual segments.
  2. To write one file, or standard input, to standard output, retaining only one segment, and discarding the others

The project took longer than I expected, and honestly I wonder how robust it is. Presently I assume every record without a time stamp is a "definition" and should be included in each split file. So these are always written. Records with time stamps are included in a given file only if the record falls between the gaps which delimit the segment. It's assumed the records are in chronological order, so if gaps are found, everything in between is written to the same file (assuming it's not discarded as a "fragment").

Thanks again to Kiyokazu Suto for doing such nice work on Garmin::FIT.

8 comments:

ben weir said...

Hey Dan,

I've run into the same problem of forgetting to reset my Garmin Edge 500 between commuting to work and commuting home.

Previously I'd used Ascent to load a .fit file and then write each lap to its own .tcx output prior to uploading them to Strava.

In searching for an easier solution I'd actually come across Kiyokazu's perl module before but never done anything with it. Glad that you took the time to write your script and post about it. I Just used it and it seemed to do the right thing!

Ben

djconnel said...

Thanks, Ben! It's encouraging when other people can get stuff to work. I've been reading through the FIT SDC docs from Garmin and there's a lot there, so I'm still refining the code a bit. For example, I'm just now putting in a check that the file type = 4 ("activity") to avoid confusion from other file types (for example, course descriptions).

ammon said...

Nice work! By now I've mostly trained myself to avoid this problem (much like learning the hard way not to forget the helmet on caltrain), but I'm sure it'll come in handy.

Krishna said...

Hey Dan,

Thanks for this-- worked quite nicely. You might want to include some very brief instructions on installing Garmin::FIT for those of us who don't normally use perl.

Krishna

djconnel said...

Thanks for the comment! I've actually got to fix these codes... the distance fields in the records need to be adjusted to account for the deleted segments, then in the lap and final ride data fields, the totals need to be adjusted. Honestly when I look at Garmin:FIT it's far from obvious how to do anything, even something as simple as adjusting distance.... so much indirection. Hopefully I can produce an updated version reasonably soon.

Really, I'd hoped Strava would adapt the same algorithm. Not yet, obviously.

Tomas Kazragis said...

Hi Dan

I've tried to use your script fit_split_on_gaps but can't split file... This is the output I get:


> perl fit_split_on_gaps.pl mar.fit
fit_split_on_gaps.pl: opening file mar.fit
mar.fit: File size: 223552, protocol version: 1.00, profile_version: 2.00
fit_split_on_gaps.pl: possible gap found of 18.78 hours
fit_split_on_gaps.pl: 2 splits found in file
fit_split_on_gaps.pl: universal: 18 records
fit_split_on_gaps.pl: split 1: 3897 records
fit_split_on_gaps.pl: split 2: 5815 records

But no fit files created. What I'm doing wrong?

Tomas

Tomas Kazragis said...

Hi Dan

I've tried to use your script fit_split_on_gaps but can't split file... This is the output I get:


> perl fit_split_on_gaps.pl mar.fit
fit_split_on_gaps.pl: opening file mar.fit
mar.fit: File size: 223552, protocol version: 1.00, profile_version: 2.00
fit_split_on_gaps.pl: possible gap found of 18.78 hours
fit_split_on_gaps.pl: 2 splits found in file
fit_split_on_gaps.pl: universal: 18 records
fit_split_on_gaps.pl: split 1: 3897 records
fit_split_on_gaps.pl: split 2: 5815 records

But no fit files created. What I'm doing wrong?

Tomas

djconnel said...

Thanks -- I noticed the latest Garmin firmware update broke this. I need to check to see if it runs with the latest update of Garmin::Fit. Honestly I've never been comfortable with the Garmin::Fit Perl module. I'd prefer to deal directly with the FIT SDK.

Dan