Bulk import with unknown data structure

Its sounds like you’ll have to write something to do the import yourself as what you’ve got sounds pretty specialised. Anything that’s existing will expect you to declare what the data is that’s being imported so it knows what to do with it.

It might sound daunting, but it’s not quite as bad as it sounds (although still a lot of work). The hardest parts that it sounds like you’ll have is sorting out what’s meant to be categories and what’s meant to be custom fields. That’s no impossible, but you’ll need to analyse the structure of your import data to see what does and doesn’t match what you expect. As an example, are categories always stored in one field, or many fields? Are those fields connected in any way? Same for the custom fields. You still need to know what’s being imported and what to do with it.

parsing the data should be a challenge. So you suggest maybe usu SQL directly to the DB?

Any ideas about the sql code?

It’s impossible to tell. I don’t know what the raw data looks like.

But it really doesn’t matter. Even to insert it into the database you’d still need to parse it into SQL format. That is made more difficult by having all of the variables that you do. Also keep in mind that if you have to add in new categories and fields then a standard SQL query isn’t going to know how to do that, so you’ll need to work around that too.

Sorry to say, but there really is no easy way to do this. Unless there is something already pre-written that can do it for you, you’ll have to do it yourself.

“unknown data structure” and “fully automate” are mutually exclusive concepts. Full automation requires a standard, known structure. However, what are unknowns to you, like field names and nesting depth, they do not necessarily mean an unknown structure. They are random values that can occur within a known structure. PHP has the ability to parse several standard structures regardless of the values contained within, such as XML, JSON, CSV, etc.

While you’re likely looking at custom import code no matter what, for full automation the source data will have to be in some standard structure. Once parsed, your custom code can figure out how to manage the varying values within.