Page Index Toggle Pages: [1] 2 3  Send Topic Send Topic Print Print
Very Hot Topic (More than 25 Replies) [Solved] Export/Import discrepancy (Read 2949 times)
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
[Solved] Export/Import discrepancy
Aug 31st, 2007 at 2:01pm
Print Post Print Post  
Solution: Problem due to field data exported from Q&A containing special characters. Cleanup of the special characters should result in a successful import.

I exported 10317 records from my Q&A database, then imported them into Sesame.  Sesame now holds 10302 records.  Where did the other 15 go?  How can I find out which 15 were lost, and why?
« Last Edit: Aug 31st, 2007 at 6:34pm by Hammer »  

**
Captain Infinity
Back to top
IP Logged
 
Bob_Hansen
Senior Member
Members
*****
Offline


WOW, They have the Internet
on computers now!

Posts: 1861
Location: Salem, NH
Joined: Nov 24th, 2002
Re: Export/Import discrepancy
Reply #1 - Aug 31st, 2007 at 2:32pm
Print Post Print Post  
Where are they ?  Why missing?
I have no answer right now, will give that some thought.

How to find them?  Here are two methods that come to mind....

1.  Sort records by unique ID in both Sesame and Q&A.  Move to Table View for both. Now check every few hundred records in order to find matches.  Record 300 should be the same value for both.  Record 600 should be the same, etc.  When not the same, go backwards by half the difference to determine where the  change happens.  Continue this as needed.

OR

2.  Export from Sesame to Ascii.  Import into empty copy of Q&A file.  Do XLU from original good Q&A for unique value in imported Q&A file.  Check XLU for value that does not exist in the imported file.  If done as a report, you can have a list of those records.  Or can do a Mass Update, selecting the records that XLU does not exist, and mark a field with "MISSING".  Now can retrieve records with "MISSING" and work on what happened, or how to export to Sesame again.

  



Bob Hansen
Sesame Database Manager Professional
Sensible Solutions Inc.
Salem, NH
603-898-8223
Skype ID = sensiblesolutions
Back to top
IP Logged
 
Hammer
YaBB Administrator
Lanticans
*****
Offline


Fire bad. Tree pretty.

Posts: 3436
Location: Ohio
Joined: Nov 22nd, 2002
Re: Export/Import discrepancy
Reply #2 - Aug 31st, 2007 at 2:51pm
Print Post Print Post  
Infinity wrote on Aug 31st, 2007 at 2:01pm:
I exported 10317 records from my Q&A database, then imported them into Sesame.  Sesame now holds 10302 records.  Where did the other 15 go?  How can I find out which 15 were lost, and why? 

First and foremost, check the text file to which you exported. Does it have 10317 lines or 10302 lines?
Also, check your log file. Does it show any errors about lines that do not have the correct number of fields?
  

- Hammer
The plural of anecdote is not data.
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: Export/Import discrepancy
Reply #3 - Aug 31st, 2007 at 3:15pm
Print Post Print Post  
Well, I powered through this.  I opened the exported file in Excel; it had 10317 lines.

Then, in Sesame, I sorted my imported records by code (it's my customer file), and got the code number to match up to the line number.  Then I just had to keep moving down the screen, watching for mismatches, and recording the missing numbers (essentially exactly as Bob suggested, though I hadn't seen his suggestion yet).  Each time, I checked the Q&A file to see if the missing records had any sort of weirdness to them, but nothing jumped out at me.  Eventually I got through all 10302 Sesame records and had a list of missing Q&A records.

Then, back to Q&A.  I exported those records, and imported them into Sesame.  Sesame accepted them all.  I now have a matching count in both programs.

But no idea why they were missed in the first pass.  I hope it doesn't happen when I export/import my invoices and workorders, because there's far more than 10,000 or each of those (approximately 3 times as many in each database, and I already know that there are missing numbers in each of them.  I've been scrupulous about keeping Customer clean, with no skipped codes, but it hasn't been a concern with Invoice or Workorder.)

Perhaps it's related to the record loss issue that Spencer brought to light?  I have not yet upgraded everything to 2.0.2, but I will before importing my Invoices and Workorders.

In any case, Customer.db is happy and full.  Time for a coffee break.  Thanks for the advice, Bob and Erika.
  

**
Captain Infinity
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: Export/Import discrepancy
Reply #4 - Aug 31st, 2007 at 3:17pm
Print Post Print Post  
Quote:
Also, check your log file. Does it show any errors about lines that do not have the correct number of fields?

The log file was error free.
  

**
Captain Infinity
Back to top
IP Logged
 
Hammer
YaBB Administrator
Lanticans
*****
Offline


Fire bad. Tree pretty.

Posts: 3436
Location: Ohio
Joined: Nov 22nd, 2002
Re: Export/Import discrepancy
Reply #5 - Aug 31st, 2007 at 3:36pm
Print Post Print Post  
Scott,

Were the missed records together, or were they scattered throughout the export file?

One thing that I have seen cause this is special characters like carriage returns or quotation marks in the data itself.

If you want to send the text file and the list of ones that got skipped into us, we'll see if we can spot any quirky data.
  

- Hammer
The plural of anecdote is not data.
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: Export/Import discrepancy
Reply #6 - Aug 31st, 2007 at 4:16pm
Print Post Print Post  
Quote:
Were the missed records together, or were they scattered throughout the export file?

Scattered.
Quote:
One thing that I have seen cause this is special characters like carriage returns or quotation marks in the data itself. 

If you want to send the text file and the list of ones that got skipped into us, we'll see if we can spot any quirky data.

Sure.  To Support@lantica.com?

  

**
Captain Infinity
Back to top
IP Logged
 
Hammer
YaBB Administrator
Lanticans
*****
Offline


Fire bad. Tree pretty.

Posts: 3436
Location: Ohio
Joined: Nov 22nd, 2002
Re: Export/Import discrepancy
Reply #7 - Aug 31st, 2007 at 4:23pm
Print Post Print Post  
Infinity wrote on Aug 31st, 2007 at 4:16pm:
Sure.  To Support@lantica.com?

Yep.
  

- Hammer
The plural of anecdote is not data.
Back to top
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: Export/Import discrepancy
Reply #8 - Aug 31st, 2007 at 6:49pm
Print Post Print Post  
Hello Scott,

The Table below has the missing ID's of the records that were not imported, what line they appeared on in the Import file, and the reason they were not imported.
Missing ID's  Record Number in CUST.ASC  Reason 
3956 3295 Caused by the value of "\", in the 26th field on record 3294, ID 10123
642 3296 Same as Above
2762 3297 Same as Above
7083 3298 Same as Above



611 4367 Caused by the value of "\", in the 13th field on record 4366, ID 9249
9250 4368 Same as Above
4368 4369 Same as Above
9228 4370 Same as Above
4370 4371 Same as Above
3552 4372 Same as Above
6844 4373 Same as Above
9253 4374 Same as Above
6817 4375 Same as Above
4375 4376 Same as Above
4376 4377 Same as Above

The \ is an escape character. So when the import sees the value \" it is interpreted as a quote that needs imported and not as a quote that comes at the end of a field before the comma. This, as you've seen, throws off the import and can cause a cascade. That is why it's important to check the fields in Q&A for both " characters and \ characters at the end of lines, as those can cause you grief in an import.

Ex. The data
"Bob","Office","\", 43.05

Would read as

Field 1 = Bob
Field 2 = Office
Field 3 = ,43.05
Field 4 =

The reason the import worked the second time, with only the missing records, is that the error was not in those records but in the ones that appeared on the line before that section of records in the original import file.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: [Solved] Export/Import discrepancy
Reply #9 - Aug 31st, 2007 at 7:08pm
Print Post Print Post  
Ray, you are amazing.  Thank you so much.

You mentioned a cascade.  Is it possible that data from one record got mixed into another because of the escape character?

Are there any other characters I should test for?
  

**
Captain Infinity
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: [Solved] Export/Import discrepancy
Reply #10 - Aug 31st, 2007 at 7:15pm
Print Post Print Post  
Oh, yeah, it's a mess.  Time for a DO OVER!!!
  

**
Captain Infinity
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: [Solved] Export/Import discrepancy
Reply #11 - Aug 31st, 2007 at 7:18pm
Print Post Print Post  
Is it just at the end of the field where the character causes a problem, or anywhere within it?
  

**
Captain Infinity
Back to top
IP Logged
 
Infinity
Senior Member
Members
*****
Offline


Diagonally parked in a
parallel dimension

Posts: 1290
Location: Massachusetts
Joined: May 27th, 2005
Re: [Solved] Export/Import discrepancy
Reply #12 - Aug 31st, 2007 at 7:27pm
Print Post Print Post  
I've checked for " at the end of a field in my Q&A database, and there are 801 records where this occurs.  The fields are text fields and hold measurements, such as 48" (48 inches).  Although those records successfully imported into Sesame, the double quote was, in every instance, changed to a single quote, so now the data reads 48' (48 feet).

Precise measurements are vital in my business.  How do I fix this?
  

**
Captain Infinity
Back to top
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: [Solved] Export/Import discrepancy
Reply #13 - Aug 31st, 2007 at 7:31pm
Print Post Print Post  
Quote:
Is it just at the end of the field where the character causes a problem, or anywhere within it?


The backslash will only be a problem at the end of a field. The Quote will be a problem anywhere in the field. What you can do is replace the " character with two single ' characters.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
Ray the Reaper
Global Moderator
Members
Lantica Support
*****
Offline


The One & The Only

Posts: 2480
Joined: Aug 20th, 2003
Re: [Solved] Export/Import discrepancy
Reply #14 - Aug 31st, 2007 at 7:36pm
Print Post Print Post  
Infinity wrote on Aug 31st, 2007 at 7:27pm:
I've checked for " at the end of a field in my Q&A database, and there are 801 records where this occurs.  The fields are text fields and hold measurements, such as 48" (48 inches).  Although those records successfully imported into Sesame, the double quote was, in every instance, changed to a single quote, so now the data reads 48' (48 feet).

Precise measurements are vital in my business.  How do I fix this?


Hello Scott,

Records that have a field value with a " in it will not import 100% successfully into Sesame unless the Quote is escaped by using a backslash. Sesame does not change the quote mark to single quotes, you may want to check the export file and see if the exporting program did.

-Ray
  

Raymond Yoxall Consulting
ray.yoxall@gmail.com
ryoxall@lantica.com
Sesame Applications, Design and Support
Back to top
IP Logged
 
Page Index Toggle Pages: [1] 2 3 
Send Topic Send Topic Print Print