Transferring data from a spreadsheet into Family Historian


Brian Horridge
 

Hi

Probably like a lot of people, I tend to keep my "raw" data in spreadsheets before inputting the details into Family Historian (maybe via Ancestral Sources) when I've identified the individual.

However, with the 1921 census I have identified from just the indexes over 1,000 entries relating to my One Name Study that I will probably want to add in when the 1921 data becomes available within an annual subscription (rather than the current pay-per-view).  Therefore I'm wondering if others have a way of transferring the spreadsheet data without having to re-type the information.

Previously, I had written a bit of code to do this but that was with an earlier version of FH when the raw GEDCOM file was much simpler.

I have thought about using a bit of code to create a very simple GEDCOM with just the new data and then using the FH  "Merge" facility to do the actual inputting - is this feasible??

Many thanks for any help / advice.

Brian


Mike Tate
 

Have you looked at the ‘Flexible CSV Importer’ plugin?

I don’t guarantee it will work for your data but it might.

The fundamental raw GEDCOM file format has not significantly changed, unless you involve Rich Text or Templated Sources.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 09:21
To: family-historian@groups.io
Subject: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Hi

Probably like a lot of people, I tend to keep my "raw" data in spreadsheets before inputting the details into Family Historian (maybe via Ancestral Sources) when I've identified the individual.

However, with the 1921 census I have identified from just the indexes over 1,000 entries relating to my One Name Study that I will probably want to add in when the 1921 data becomes available within an annual subscription (rather than the current pay-per-view).  Therefore I'm wondering if others have a way of transferring the spreadsheet data without having to re-type the information.

Previously, I had written a bit of code to do this but that was with an earlier version of FH when the raw GEDCOM file was much simpler.

I have thought about using a bit of code to create a very simple GEDCOM with just the new data and then using the FH  "Merge" facility to do the actual inputting - is this feasible??

Many thanks for any help / advice.

Brian


Brian Horridge
 

Mike

I've just had a quick look at that plugin and it seems as if it might work for what I want to do - I'll download it and have a play.

On a quick glance I can relatively easily write a bit of code to convert my spreadsheet into a format for the CSV file (headers etc)

I do have one question - one of the Plugin store descriptions says "link to individuals already present in the Project using their Custom Id" - does this mean that I have to use the Custom ID field or can I used the normal FH identifier number

Many thanks

Brian

On 03/03/2022 10:27, Mike Tate wrote:

Have you looked at the ‘Flexible CSV Importer’ plugin?

I don’t guarantee it will work for your data but it might.

The fundamental raw GEDCOM file format has not significantly changed, unless you involve Rich Text or Templated Sources.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 09:21
To: family-historian@groups.io
Subject: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Hi

Probably like a lot of people, I tend to keep my "raw" data in spreadsheets before inputting the details into Family Historian (maybe via Ancestral Sources) when I've identified the individual.

However, with the 1921 census I have identified from just the indexes over 1,000 entries relating to my One Name Study that I will probably want to add in when the 1921 data becomes available within an annual subscription (rather than the current pay-per-view).  Therefore I'm wondering if others have a way of transferring the spreadsheet data without having to re-type the information.

Previously, I had written a bit of code to do this but that was with an earlier version of FH when the raw GEDCOM file was much simpler.

I have thought about using a bit of code to create a very simple GEDCOM with just the new data and then using the FH  "Merge" facility to do the actual inputting - is this feasible??

Many thanks for any help / advice.

Brian


Virus-free. www.avg.com


Mike Tate
 

Brian, there are a number of options.

 

You could use the plugin to populate an entirely new Project.

I strongly advise you to do so initially anyway, to iron out any issues with the CSV.

Then that new Project can be File Merged with your master Project and should auto-sync using Individual Names and other details.

 

As it says in the detailed Help under Running the Plugin:

“To sync with an established Project, use the FH Tools > Record Identifiers… option to copy Record Id to the Custom ID and use those numbers as the REFN values.”

 

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 13:43
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike

I've just had a quick look at that plugin and it seems as if it might work for what I want to do - I'll download it and have a play.

On a quick glance I can relatively easily write a bit of code to convert my spreadsheet into a format for the CSV file (headers etc)

I do have one question - one of the Plugin store descriptions says "link to individuals already present in the Project using their Custom Id" - does this mean that I have to use the Custom ID field or can I used the normal FH identifier number

Many thanks

Brian


Brian Horridge
 

Mike

Thanks - I'll have a play and see how it works.  Also thanks for clarifying about the Record ID / Custom ID.

Brian

On 03/03/2022 14:51, Mike Tate wrote:

Brian, there are a number of options.

 

You could use the plugin to populate an entirely new Project.

I strongly advise you to do so initially anyway, to iron out any issues with the CSV.

Then that new Project can be File Merged with your master Project and should auto-sync using Individual Names and other details.

 

As it says in the detailed Help under Running the Plugin:

“To sync with an established Project, use the FH Tools > Record Identifiers… option to copy Record Id to the Custom ID and use those numbers as the REFN values.”

 

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 13:43
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike

I've just had a quick look at that plugin and it seems as if it might work for what I want to do - I'll download it and have a play.

On a quick glance I can relatively easily write a bit of code to convert my spreadsheet into a format for the CSV file (headers etc)

I do have one question - one of the Plugin store descriptions says "link to individuals already present in the Project using their Custom Id" - does this mean that I have to use the Custom ID field or can I used the normal FH identifier number

Many thanks

Brian


Virus-free. www.avg.com


Brian Horridge
 

Mike

I've been playing with the Plugin and, generally I've got it to do what I want but I'm having a problem adding a note to the census occupation (the 1921 employer which I want to copy into a note in the Occupation event).

I've tried using the CSV file column header "P.OCCU.NOTE" but all it seems to do is create a second Occupation event without a date but with the required text in a note (and also adds the same Occupation source).

I can get the plugin to add the Note to the Occupation Source (using P.OCCU.SOUR.NOTE) but I'd prefer it against the actual Occupation rather than the Source

Thanks

Brian

On 03/03/2022 10:27, Mike Tate wrote:

Have you looked at the ‘Flexible CSV Importer’ plugin?

I don’t guarantee it will work for your data but it might.

The fundamental raw GEDCOM file format has not significantly changed, unless you involve Rich Text or Templated Sources.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 09:21
To: family-historian@groups.io
Subject: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Hi

Probably like a lot of people, I tend to keep my "raw" data in spreadsheets before inputting the details into Family Historian (maybe via Ancestral Sources) when I've identified the individual.

However, with the 1921 census I have identified from just the indexes over 1,000 entries relating to my One Name Study that I will probably want to add in when the 1921 data becomes available within an annual subscription (rather than the current pay-per-view).  Therefore I'm wondering if others have a way of transferring the spreadsheet data without having to re-type the information.

Previously, I had written a bit of code to do this but that was with an earlier version of FH when the raw GEDCOM file was much simpler.

I have thought about using a bit of code to create a very simple GEDCOM with just the new data and then using the FH  "Merge" facility to do the actual inputting - is this feasible??

Many thanks for any help / advice.

Brian


Virus-free. www.avg.com


Mike Tate
 

Brian,

You will need to give me a few more clues.

Are you creating a new Project or adding to an existing Project?

If I use P.OCCU.NOTE it adds the text to the Occupation fact Note.

How are you associating the new text with the desired Occupation fact?

The plugin uses Date, Place, etc, fields to sync new data to existing data.

i.e. To use an existing Occupation fact with a Date of 19 June 1921 then your CSV data must have a P.OCCU.DATE column with 19 June 1921.

If the plugin cannot associate the CSV data with any existing fact it creates a new fact instance.

 

I see that the Help page does not mention such synchronisation concepts, so I will update it.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 13:11
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike

I've been playing with the Plugin and, generally I've got it to do what I want but I'm having a problem adding a note to the census occupation (the 1921 employer which I want to copy into a note in the Occupation event).

I've tried using the CSV file column header "P.OCCU.NOTE" but all it seems to do is create a second Occupation event without a date but with the required text in a note (and also adds the same Occupation source).

I can get the plugin to add the Note to the Occupation Source (using P.OCCU.SOUR.NOTE) but I'd prefer it against the actual Occupation rather than the Source

Thanks

Brian

On 03/03/2022 10:27, Mike Tate wrote:

Have you looked at the ‘Flexible CSV Importer’ plugin?

I don’t guarantee it will work for your data but it might.

The fundamental raw GEDCOM file format has not significantly changed, unless you involve Rich Text or Templated Sources.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 03 March 2022 09:21
To: family-historian@groups.io
Subject: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Hi

Probably like a lot of people, I tend to keep my "raw" data in spreadsheets before inputting the details into Family Historian (maybe via Ancestral Sources) when I've identified the individual.

However, with the 1921 census I have identified from just the indexes over 1,000 entries relating to my One Name Study that I will probably want to add in when the 1921 data becomes available within an annual subscription (rather than the current pay-per-view).  Therefore I'm wondering if others have a way of transferring the spreadsheet data without having to re-type the information.

Previously, I had written a bit of code to do this but that was with an earlier version of FH when the raw GEDCOM file was much simpler.

I have thought about using a bit of code to create a very simple GEDCOM with just the new data and then using the FH  "Merge" facility to do the actual inputting - is this feasible??

Many thanks for any help / advice.

Brian

 


Brian Horridge
 

Mike

I'm adding the census event data to an existing GEDCOM file which does not have a 1921 Census or Occupation for the relevant individual so both Census and Occupation are new events / attributes

I have 1 row of data per individual for both the Census Event and the Occupation Attribute and CSV column headers for the Occupation are as follows :-  (the Census event columns work OK as does the REFN ID number) so I'm ignoring them here)

P.OCCU P.OCCU.DATE P.OCCU.PLAC P.OCCU.AGE P.OCCU.NOTE P.OCCU.SOUR>TITL P.OCCU.SOUR.PAGE P.OCCU.SOUR.NOTE
Lighterman 19 Jun 1921 Mile End 40y 11m
1921 Census RG15/1671-223 C Crawley Waterman  370 Commercial Rd - River Thames

The above row adds the data as a Note to the Source  as follows:-



P.OCCU P.OCCU.DATE P.OCCU.PLAC P.OCCU.AGE P.OCCU.NOTE P.OCCU.SOUR>TITL P.OCCU.SOUR.PAGE P.OCCU.SOUR.NOTE
Lighterman 19 Jun 1921 Mile End 40y 11m C Crawley Waterman  370 Commercial Rd - River Thames 1921 Census RG15/1671-223

The data above this time creates two Occupation attributes - the first with the Occupation, Date, Place, Age and Source and a second with just the same Source and a separate Occupation Note as below (without a date):-



I'm trying to get the Note from the 2nd example to appear with the first (and only Occupation).

From what you said, could the Plugin be creating the first Occupation with the occupation / age etc and then, because that now exists, it creates a 2nd Occupation with just the Note ??.  As I said, all the data is on the same (single) row of the CSV file as shown in the examples above

Hope that explains my problem.

Many thanks for your forbearance.

Brian

On 04/03/2022 14:53, Mike Tate wrote:

Brian,

You will need to give me a few more clues.

Are you creating a new Project or adding to an existing Project?

If I use P.OCCU.NOTE it adds the text to the Occupation fact Note.

How are you associating the new text with the desired Occupation fact?

The plugin uses Date, Place, etc, fields to sync new data to existing data.

i.e. To use an existing Occupation fact with a Date of 19 June 1921 then your CSV data must have a P.OCCU.DATE column with 19 June 1921.

If the plugin cannot associate the CSV data with any existing fact it creates a new fact instance.

 

I see that the Help page does not mention such synchronisation concepts, so I will update it.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 13:11
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike

I've been playing with the Plugin and, generally I've got it to do what I want but I'm having a problem adding a note to the census occupation (the 1921 employer which I want to copy into a note in the Occupation event).

I've tried using the CSV file column header "P.OCCU.NOTE" but all it seems to do is create a second Occupation event without a date but with the required text in a note (and also adds the same Occupation source).

I can get the plugin to add the Note to the Occupation Source (using P.OCCU.SOUR.NOTE) but I'd prefer it against the actual Occupation rather than the Source

Thanks

Brian




Mike Tate
 

Thank you for the details. I will have to investigate exactly why that is happening.

However, if you swap the P.OCCU.AGE and P.OCCU.NOTE columns it seems to work correctly.

 

BTW: If P.CENS.AGE and P.CENS.NOTE columns exist then the Census Event suffers the same way.

Presumably, you don’t have both?

 

Mike


Brian Horridge
 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 18:17:22 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

Thank you for the details. I will have to investigate exactly why that is happening.

However, if you swap the P.OCCU.AGE and P.OCCU.NOTE columns it seems to work correctly.

 

BTW: If P.CENS.AGE and P.CENS.NOTE columns exist then the Census Event suffers the same way.

Presumably, you don’t have both?

 

Mike


Mike Tate
 

I have not checked every fact, but the workaround for OCCU works for CENS, and I suspect for all other facts too.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:30
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian


Brian Horridge
 

Thanks - I'll experiment later.

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 18:53:48 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I have not checked every fact, but the workaround for OCCU works for CENS, and I suspect for all other facts too.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:30
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian


Mike Tate
 

I forgot to mention that the more standard convention is to add the AGE to the CENS and not the OCCU because the Age is on the Census record.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:55
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Thanks - I'll experiment later.

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 18:53:48 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I have not checked every fact, but the workaround for OCCU works for CENS, and I suspect for all other facts too.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:30
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian


Brian Horridge
 

Mike
I do have the age in the census event as well as the census occupation (ie copying the way Ancestral Sources does it though not really sure why!!).
Brian

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 19:22:53 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I forgot to mention that the more standard convention is to add the AGE to the CENS and not the OCCU because the Age is on the Census record.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:55
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Thanks - I'll experiment later.

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 18:53:48 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I have not checked every fact, but the workaround for OCCU works for CENS, and I suspect for all other facts too.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:30
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian


John Elvin
 

I've just looked at data added by AS for a census and it doesn't have the age on the occupation. What I do see in the facts tab of the property box is the calculated age in brackets.

John

On 04/03/2022 19:29, Brian Horridge via groups.io wrote:
Mike
I do have the age in the census event as well as the census occupation (ie copying the way Ancestral Sources does it though not really sure why!!).
Brian

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 19:22:53 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I forgot to mention that the more standard convention is to add the AGE to the CENS and not the OCCU because the Age is on the Census record.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:55
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Thanks - I'll experiment later.

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 4 March 2022 18:53:48 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

I have not checked every fact, but the workaround for OCCU works for CENS, and I suspect for all other facts too.

 

From: family-historian@groups.io <family-historian@groups.io> On Behalf Of Brian Horridge via groups.io
Sent: 04 March 2022 18:30
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

 

Mike
I'm glad it's not just me!!
At the moment I do not have any census data that I'd want to put in  a census event note.   However, I had considered using it for the Orphanhood value which I could later use to check for dead parents so I'd be glad if you can fix that as well.
Many thanks
Brian


Lorna Craig
 

It depends on the settings you choose in AS.    In the AS Tools>Options>Census Settings under Occupation Facts there is a tick box for 'Record individual age in occupation'

Lorna

On 04/03/2022 19:41, John Elvin wrote:

I've just looked at data added by AS for a census and it doesn't have the age on the occupation. What I do see in the facts tab of the property box is the calculated age in brackets.

John

On 04/03/2022 19:29, Brian Horridge via groups.io wrote:
Mike
I do have the age in the census event as well as the census occupation (ie copying the way Ancestral Sources does it though not really sure why!!).
Brian


Nick Walker (Ancestral Sources)
 

The reason why AS doesn't save the age to the occupation fact by default is because the age on a census record is often not accurate. The age given is related to the census record and is therefore recorded in the census fact and recording it there has the clear implication that it may or may not be accurate as this is the age given in the census. There's no need to record that possibly inaccurate age in the occupation too - the age is a feature of the census, not the occupation. But as Lorna said, there is an option to include the census age in the occupation if you don't agree with my logic!

All the best

Nick
Ancestral Sources Developer

On 04/03/2022 20:20 Lorna Craig via groups.io <l.m.craig@...> wrote:


It depends on the settings you choose in AS.    In the AS Tools>Options>Census Settings under Occupation Facts there is a tick box for 'Record individual age in occupation'

Lorna

On 04/03/2022 19:41, John Elvin wrote:

I've just looked at data added by AS for a census and it doesn't have the age on the occupation. What I do see in the facts tab of the property box is the calculated age in brackets.

John

On 04/03/2022 19:29, Brian Horridge via groups.io wrote:
Mike
I do have the age in the census event as well as the census occupation (ie copying the way Ancestral Sources does it though not really sure why!!).
Brian


Mike Tate
 

Brian,

I have investigated the plugin and discovered why it was not handling the P.OCCU.AGE and P.OCCU.NOTE fields correctly.

It will be updated in the Plugin Store next week.

Mike

 


Brian Horridge
 

Mike
Hope it did not cause too much work but thanks very much for all your hard work on our behalf.
I have been playing with this plugin and it looks very useful for other spreadsheet transfers.
Again, many thanks
Brian

Sent via BT Email App



From: Mike Tate <post@...>
Sent: 5 March 2022 18:52:38 GMT+00:00
To: family-historian@groups.io
Subject: Re: [family-historian] Transferring data from a spreadsheet into Family Historian

Brian,

I have investigated the plugin and discovered why it was not handling the P.OCCU.AGE and P.OCCU.NOTE fields correctly.

It will be updated in the Plugin Store next week.

Mike

 


Vyger
 

Brian, I often did this in the past using VBA within Excel to beef up the data in my four ONS's although I have not delved into the FH Gedcom requirements or possibilities. I was able to grab the full information from the Irish 1901 census as a good starting point, the 1921 has little interest to me as Ireland north or south are not included.

I think this is well below your level of expertise but I did post some starter videos back in the day facilitating Gedcom import into Rootsmagic. It was a very flexible way of getting large datasets into a genealogy program, I have not tried the FH plugin but will do out of interest.

https://youtu.be/HWWu-eq4B0M