Geospatial Studio: Jumpstarting Metadata Creation with the NC GIS Metadata Profile

[Vogler] My name is John Vogler. I’m a researcher here at the
Center for Geospatial analytics, and I’m going to quickly turn it over to Jeff Essic. He’s a research
librarian at NC State, with a special love for GIS and mapping. And he’s going to be introducing
our instructors today. Welcome. [Essic] Thanks, John. I just want to say a few
things to start us off. The way this kind of came about–I’m on a metadata
committee that the state has been working on, creating a metadata profile over the last several years. So these
three here are also on committee. And so I said we really need to have a workshop at NC State on metadata, it would be very valuable. So, we have Dr.
Tim Mulrooney from NCCU and Sarah Wray from NCDOT, and Chelsea Duncan also from NCDOT. I want to tell you two very short stories about metadata: one is that I remember as a
kid–and probably about you do too–my grandmother and my mother writing on the back
of photographs the names of who was in the photograph. It was very common, and they spent a lot of
hours doing that, and they’re both passed away now, but at the time I remember, you
know, making jokes with them, saying, “Why are you writing, you know, Uncle Virgil’s name
on the back of this picture? Everybody knows that’s who that is.” And you know, she said, “One day you’re going to want to have this written down.” I
guess that’s now. I very much now appreciate the work that they did. I
think that that’s metadata–when you write stuff like that on the back of a
picture. And then the other story is just how much I came to appreciate metadata
when I started working at the library – and especially machine readable metadata, because
I started feeding the catalogue a lot of data that we were receiving and so
anything that came in with an XML file or something that was machine readable so
I could some batch processing and put that into our catalogue that became very useful. And
that’s when it really struck me just how valuable the metadata is–not
just for internal purposes but for other people looking for data. So that’s one of
the reasons I really advocate and encourage people to create it and use it. All
right. Turning it over to you folks. Thanks again for
being here. [Mulrooney] My name is Tim Mulrooney, a professor at North Carolina Central University.
I got my start in metadata working for the military in an environmental program about
15 years ago and we were asked to start to create maps and then some one at
Pentagon basically asked us where to get these data from and previous people
didn’t have any information and well the Pentagon don’t like to hear the words “I
don’t know” so we have to go we developed data GPS, visit all these places in
Kentucky or Puerto Rico, you want all these other places you know and redevelop
these data. so starting in my office I became our metadata expert, and then I
started doing — I was talking to Kelsey about it– I used to programmatically access EPA elements
because my boss would say you know how many elements were created before 2002
or how many you know purposes weren’t auto-populated. It’s really hard to go
through and eventually click on each entry and toggle through each individual
element so I started to write this too starting to write this to Excel and do
access you know do information on that when I back went back for my PhD I had
an idea to do metadata so probably one of the dissertations on metadata and I
mentioned it to Ann Paine who was on the committee before and she volunteered me
for this committee probably about four or five years ago when I got so you know I kind
of have a bit of a love hate relationship so with metadata it’s all
kind of introduced everyone here Craig not gonna be here Craig is our is our
graduate student. He had a family emergency but we have a grant
through the NCDOT research program to provide metadata training to the state
so help us on the assets, videos, training that we’re developing right now and
this is part and parcel of that. and [Wray] Hi, I’m Sarah Wray. I’m with the
department of transportation GIS unit the spatial data manager there. I’m actually
chair for the state metadata committee implementation. So what that means is
that well the three of us will show you a couple slides but basically it’s the
coming to you guys to tell you about the metadata and do the training. That’s why I’m here. [Chelsea] I’m Chelsea. I work for Sarah with the spatial data management group with NC DOT. I was on the original committee to develop the standard and am now serving on the committee under Sarah as well. Been working on it for about 4 years now. I bring a lot of technical expertise to the committee. And I’m here if you have any questions. [Mulrooney] So, it looks like there are a lot of people here who have experience
with metadata, heard of it, probably have some love-hate relationship to it. What do people work with metadata on a regular
basis. I try to it’s hard you know usually metadata you know and all of our
books is usually have the very last chapters I’m always bustling we’ll get
to that you know right before and it was before Christmas or whatever so even in
my classes metadata is usually it kind of looked at you know at the very end of
you know everything here so you know in terms of our metadata relationship the
North Carolina Geographic Information Coordinating Council– basically
the goals are to facilitate metadata creation, improve data quality,
so I’ll talk about quality measures that some of the things that I like to focus
on because they’re quantitative in nature. We’re looking the individual 400+ elements, integrate these ISO standards, so we’ll talk about the actual
profile. I’ll pass it out we’ll kind of look at a little bit, and preserve data
investments and basically enhance data discovery and sharing. yYou know and I
always send my students to North Carolina One Map because you can type
in keyword, spatial extent, and you can find pretty much anything that you want.
So I can tell them go download schools, go look up swine lagoons or whatever and
they can download it and because their metadata is so rich. So we talked about
who created profile. I’ll let Sarah explain that. [Wray] Who created the
profiles? So when the North Carolina Geographic Information Coordinating
Council starts to put together committees they always populate those
committees with representation from the rest of the committees, the
subcommittees, to the council. So we always have local government, state
government, federal, university system if we can, private sector if appropriate,
and professional associations if we can get the membership. In this case for this
metadata profile what we were doing is we had an approved metadata standard,
geospatial metadata standard for the state.
Well that standard was being put to sleep, and we were going with a new
standard, and so we needed to decide how to interact with a new standard for the
state.So that’s why we created a subcommittee to make the recommendation
back to the state Mapping Advisory Committee, back to the NCGICC. So who was
on there? City of Greensboro, Lee County, Wake County, we actually had membership
from Canada–they are just a little bit ahead of us in the profile generation and
the interaction with the ISO standards, so they were actually really
helpful because they’d already suffered some things, we were able to learn some
of their lessons. State government, we have CGIA, transportation,
cultural resources. Cultural resources are really important because they are
responsible for the archiving for the state and all the archiving rules, and
there’s a lot of places in the metadata to make those recommendations and we
wanted to make sure that we were at least in alignment with the state archiving rules if possible. US Fish and
Wildlife, the FGDC actually was partnered with us and also helped facilitate some
questions and answers as they were moving toward a new standard adoption. NC
State and you met Jeff, NC Central– Dr. Tim as I call him, private sector
GeoMaxim. We were able to hire probably one of the top, within the top
five if we’re not going to say the top, metadata expert in the country to
serve an original committee and help staff the committee, and then we had
membership from URISA because they had vested interest at the local and the
government level to make sure we got all the information out there. [Mulrooney] Pretty much standard uses your ISO
standards your nineteen 105 in nineteen won won 5-1 your 19119 the standard
we’ll talk about today also looks or addresses contents of
geospatial metadata compliancy. Typically when orbiting Arc Catalog
there’s ways that you can look at all these and we’re an ESRI shop, you know UNC system, and so our default browser our metadata editor is going to be
ESRI for the most part, but this previous standard was adopted in 1996 and we’re
kind of currently migrating the new one which was adopted December 30th 2016 so
we’ll just talk about some of the changes and some of the difference is to
make your data a lot more discoverable and ways that you can address it. It
applies to broad range of geospatial resources, data collections,
basically you know services and other things that didn’t exist when the CSD GM
was created late 1990s it’s a flexible easily customizable and it’s
standardized across multiple communities So this is the North Carolina state local government profile. Within
the profile when you receive data from an organization one of things that you
need to know I’m sure a lot of people know this one with this these are
parcels for Camden County so people taking intro level GIS we have
any folks here who created the data all this information what time period of
these data are from their standardization the entries for these
are going to be standardized what these actually represent and you can see
these represent parcel how did you create these data when were these data
published what software did you use to create these who created the information about
the datasets so there’s going to be different data contents for some
metadata context so like I said what well there’s more than 400 individual entries
that you can type in or enter in or can be populated for the particular entries
so it’s going to be a lot different than you know like a you know Welker resource
that you put at the end of a research paper or the distribution or news
constraints so you know each organization has some
really good verbiage as to what they’re used to distribution constraints are. The
military is very specific about the wording for these who’s in charge of
distributing the data our updates required how often how can I get in
touch with a person email address or phone number what are the attributes for
these data then were the types of the end of the types and we look at the
profile and I’ll actually pass out the profile here lt’s a little harder to read with black and
white but when you go to profile at the NC One Map website to metadata see the
green elements are gonna be your required elements so elements such as the title,
publication date, custodian, online linkage, the abstract and this profile gets very
specific as to the different types of entries and also has the main table for
each of these–the use constraints, all the categories which is important with
length of the new ISO standard geographic extent to metadata creation
date point of contact and then there’s some fixed content with the days that
language encounter coding these are optional elements so we have got a
purpose, currentness reference, metadata standard name, process date
completeness so we’ve got required elements we’ve got suggested elements
you know PDF document goes and talks about all these possible
values for all of these and then main table back and later we’ll have a
little hands-on exercise well you know just kind of ask questions in ways to
you know understand or possible values The profile could be intended for state
government, local government, federal government my class is basically
described it’s going to be for everyone who uses data for with about North
Carolina and GIS data there’s some standardization. Like Sarah mentioned
before we work with international organizations Alberta Canada and there’s
some relationship with the Republic of Namibia. [Wray] Someone from the NC OneMap website they got a request from the government of Namibia to use the profile that we created because it was
very local- and state-government-based. They really liked the way that we broke down
the profile and as Tim will get to how we maneuvered through it so they asked us
permission to basically use the profile that we generated. we didn’t know what to
do with that. It came back to the committee. It came all the way down to the metadata committee and were
like yes please use it–it’s publicly available they don’t really need to ask
permission because it’s on the website but that’s actually kind of cool. So we
ended up making the recommendations back up through the committee cycle and NCGICC wrote an official approval letter and data sharing letter to the government of Namibia for the use of the profile. We actually just asked their value to
let us know how their implementation went. So of the government red tape, it was like “Is this real?” So
we actually had to do a bunch of research and make phone calls
it’s to make sure that it wasn’t you know … but it ended up
being an honest-to-goodness request. It ended up being approved and a government document signed off on it. So we include them on the slide because that’s
pretty cool. [Mulrooney] And basically pretty much we encourage everyone and you know at least
the students I work with you know whenever they develop data I encourage
them to make their data discoverable you know and so you just repeat or you know
try to prevent any of the duplication of resources especially if data already
exists out there. Any questions about any kind of a business case for metadata? We gave this workshop at NCGIS before, we kinda had to stop a bit. When you actually look at the profile
I’ve givien out the profile this is a link to the profile so North Carolina
One Map dot gov and then left-hand navigation tab will see metadata the
business case for metadata you can go download these there’s templates that you can
download for teachers and different software applications too. If you look at the document we’ve got acknowledgments, this basically gives a great history of it, value of compliant metadata, the standards how it
was built for pretty much the ISO standards combine … then we have combination of our geospatial data we distinguish those
from our geospatial services and we have our code list reference documents XPath
and our feature catalog that describes some of our … When I broke this
down because I’m looking to present this next week at another conference in
Winston-Salem you know for intro GIS users I wanted to kind of just break it
down into these are some of the examples of the profile these are kind of a big
big points here and typically for the publication date it’s going to be
required and the format is going to be in a couple different types of formats
year year year year with the dashes or without if it’s not known it has
specifications for those an abstract is required so as you read this document
read these tables these are some of the big bullet points that we have the status
is required and there’s only possible values like historical archive required
planned ongoing completed under development and obsolete the topic
category’s taken from one the ISO standards and it could be one of 23
possible values so it’s required our maintenance frequency is required to be
one of only twelve possible values use constraints is required a
process description is required as a free text entry and so these are some of the big
bullet points for geospatial metadata and then you differentiate those from
geospatial services so the metadata scope needs to be service and the
service type must be populated right now we’re working on the code lists for
those and the online function codes required from a domain of one
of five possible values of the people download information offline access
order or search so this standard differentiates between geospatial data
and services which you know your CS DGM obviously doesn’t when you look at the table and this is pretty much a breakdown of the table we’ve got the element right here we’ve
got a data type and then what Chelsea and Craig went through and did they went
through and put in examples of best practice for a possible
title so they went and populated all these you know best practices what’s a good
example of a title [Sarah] yeah that was really important to the committee it’s
not just saying put a title in there right it’s hard enough to tell people to
document their processes in their work but the least we could do is show them what
title looks like what we would say a good title is we actually had
discussions about things like spell out the acronym in your title
don’t have in your title everything just alphabet soup that you can
reference it later on but these are kind of discussions that they committee had
for example so we spend a lot of time one defining what understanding the
international standard and how we wanted it to look as implementations in North Carolina So did we choose those elements on purpose yes
the datatype comes straight from the standard element comes straight from
standard but the recommendation implementation of those examples are a
lot of what the previous committee did so where we really hope that that helps
[Mulrooney] yeah I think it’s important because when I open up my metadata I don’t like to
look at everything I just put item description which only had five
different elements or you can open it up in Arc Catalog right mouse click data
and click on item description I want to go through everything so title of one of
those elements in your description so we’ve got element name which is going to
be mandatory because it’s in green types of values allowed or free text and
then just talks about the guidance and examples for each of these so and
we’ve got a number of different entries here I’m not going to go through all of
these and try to get some of some of the points there the metadata scope and if
you open up this PDF document from the web click on this scope and
this will take you to the link to all the possible values for scope this is
for geospatial services so the scope can only be service for the service
type it can be free text so like I mentioned before the different types and the different types of services that you can turn on there so
we differentiate between data and services so we’ve got a hyperlink code
to lists that gives us a set of it set of values and you can see that in the appendix at the end [Wray] A lot of our metadata editors those are
already programmed in and they’re just your dropdowns but in the profile itself
we hyperlink it an appendix so you can see and pre-plan if you
need to but most of the editors it’s pre-programmed for you you just have to
choose [Mulrooney] we have a code list you know they’re linked so they basically give us
the metadata consistency search and discovery and that in turn is
international transformation … there’s only one of twenty three or twenty
one possible values that you could have so if it’s something that’s illegal
then you know it’s something that’ll be flagged when it’s checked later we have different code lists so the topic category you
can see farming and have examples of each so we’ve got a description
for examples of feature classes or themes … one of them this is just service
this is required service that’s the only option for geospatial services for metadata scope and then we’ve got other related
documents the best supporting documents ISO pretty much as you can play up those
standards but it’s got CSD GM those are elements so we have a standard
publication for that and then we’ve got good metadata resource via FGDC
we ran a boot camp or workshop for metadata in May and we really got in
depth with some of the metadata and have a lot of good resources it’s getting a little obsolete because it still references a lot of CSDGM and then there’s going to be other
resources available so right now tool that will export certain metadata elements
spreadsheets you can look at some of the state profile compliancy
using some Python code …and we have XPaths at the end so you can automatically access
each of the metadata elements and those of you know those are some of the things
that I’m starting to work on with Python exporting or accessing these and writing
these or checking these particular values so we can look at the Xpaths and
those Xpaths at the end these Xpaths are are different based on your standard and in
our catalog every second reference or catalog it has a number of different
standards it’s got the EPA and when we were at metadata bootcamp we played with the EPA metadata editor
which checks for compliancy you know flag or highlight some of the non-compliant
entries looks at the 91 39 the 91 1 5 and CSDGM and even inspire … standard
and the North American profile so there’s a lot of ways that you could
view those export those using various translators so it’s got some of the
different XPath so you programmatically access [Wray] And check with XPath before we get into the activity the
international standard is UML based we didn’t feel as a committee that it would
be worth our time to try to train data editors but particularly the local
government level what UML is and how to break down that
kind of IT design of that notation so where are the Xpaths come in and is
actually if you understand UML of the ISO standard it helps you find where the
location and the actual standard is for that information so the xpaths are it is
the nested almost folder structure if you will of where that metadata information lives if you’re not going to be
programming against it and you’re just doing your editing in a normal metadata
editor you may not even ever care about the Xpaths that’s why it’s the end of the document
but if you are trying to program against it find your way through the depths of
the XML this is the way to find your it’s how to find your way back home
that’s why it says breadcrumbs so it tells you the structure within the XML
where to find installments that you’re dealing with it is kind of advanced
interaction with metadata but this is the kind of community I want to talk
about those kinds of things with you guys are the kinds of people that would
get into the UML understand how it’s embedded and being able to interact with
it in different ways than just simply from an editor client first level so I’m
glad he brings it up and that’s kind of the stuff we ‘re talking about there. [Mulrooney] What we’re going to look at in a little bit here is we’re going to
just create the standard so how many questions we have eight different
questions look at these but wanna look at you know which of the
following acceptable role code for the responsible party so as we kind of look
at the actual standard I like to do it in my PDF and just hit control F just go
through and look for the word role code this is what the actual standard looks
like I can type in the word role code and navigate and then when you click
on the actual document pick them in and these are the possible values for role codes
and explained exactly what each of these are so we’ve got a custodian that
accepts accountability responsibility for the data and care and maintenance of the
resource or the role of a party who can be contacted for acquiring knowledge
about or acquisition of the resource so you can see these interactive ways to access this which is the
following what coordinate format is required for that geographics extent bounding box so we can look for the word bounding box in the actual document at the top of page 11 this is usually going to be in latitude and longitude … are place keywords required … are they optional [Wray] and one of the reasons why we wanted to
still acknowledge the old FDGC standard is because at the time FGDC was
still approving still endorsed all of those standards at this
time they are making the recommendation and making the steps to move forward to
move away from CSD GM and just endorse the international standard
so as the federal government makes those leaps we’ll be making those leap and
things that are optional because they’re CSD GM we may be dropping from the list
depending on how the committee feels about those particular items may just leave them
optional as normal optional not optional because they were CSD GM. So that’s some of the activity of the commitee [Audience member] this a place keyword brings up a
question do you know how people are handling this for state-level datasets for example North Carolina … or do they try to fill in other counties and towns [Wray] I don’t know I think it’s more applicable by dataset so when you’re talking with the
they’re now called DEQ right departmental of
environmental quality they used to be the DNR right so they kind of operate their
their you know their boundaries are the river basins themselves so their place
keywords may be kind or organized based on they datatypes they’re dealing we I don’t think we make a
recommendation for all of the counties our recommendation at DOT is just the
state and sometimes the district divisions our personal break downs
of those locations but we’re not we don’t have a habit of putting all 100
listing all 100 counties for the place um keywords but it just takes up a lot of space people can scroll on by so that
being said we have regularly published products that all 100 counties plus Band
of Cherokee and such have to be in the drop-down list so you have pages and
pages of metadata attribute domain stuff that includes them all not necessarily in the place keywords but its
internal to DOT what we’ve been discussing data content data Enterprise
content data management wise is what it’s an applicable advice key word that
people can still find what you’re looking
so that’s really what the recommendation is if you want people to find
your stuff maybe you don’t the idea is that you do if you want people
to find your stuff put as much in there so that they if that’s what you think
people would search for what you’re producing populate it that way [Mulrooney] which of the following elements required for documentation for geospatial services pretty sure it’s
the feature catalog … just got a couple other ones what topic category best applies to zoning maps what is the cost of the CSDGM pretty sure it’s free right I know the ISO is a hundred or something dollars
I don’t know what is the preferred online linkage XPath so we get a little further into the weeds here this has
got the XPath for my online linkage right here so for the metadata metadata
distribution you can see the UML implementation of it looks a little different than your XML feature catalog is used to
document and this just talks about the entity and attribute information here – I’m doing a workshop next week for you know beginners where we’re going to read through this… any
questions about you know this actual document how to read it how to interpret it we just got a couple of content overview
and then we’ll talk about some resources when we write a good title I think we mentioned this before what are what entails a good title try to avoid
acronyms abbreviations that are not commonly understood through a file name
or other identifying information so let’s talk about the file name the when
the where date within this particular title this was important especially if I start
to go and you know try to use Python to programmatically access these so you
know I need to find ways to account for years you know what make sense you know I’m gonna parse out of different years That year either better start with a 19 or 2
or otherwise it’s going to be kind of you know I’m going to flag it as incorrect so
these are the different ways that it could be you know that can be
represented here according to the standard or responsible party like we
mentioned before there’s a custodian or point of contact is required we’ve got
additional responsible parties are optional they include vendors who
created it and entities that distributed individuals or agencies that have processed or got their hands on so you can see a lot of different
information theme keywords so it’s got a robust set of descriptive theme-related
keywords such as wetlands salinity and I use one more of the ISO topic categories and I’ll explain those in a little bit and the the CSDGM theme keywords should also include an ISO topic information the use constraints indicate any
restriction associated with using the data having worked with the military
there’s always a lot of use of constraint distribution constraints
attached to these it’s a lot longer than this I know we were talking with someone
at one of the folks today who was had to go to the lawyers and put together
some big verbiage on which the use of distribution constraint should be for their particular data so
just be aware of you know they typically talked about no warranties
expressed or implied regarding the accuracy of the data I’ll mention a
couple things about data quality which get into my specialty [Wray] also as you do in
research make sure you’re not relying upon something you can’t use for that so
that’s a place to go to look and if you see it’s kind of ambiguous go to that point
of contact and double check the last thing you want is to be doing your research and not be able to include something or be able to report out
on it or be informed by it and just make sure you you know what you’re dealing
with as well [Mulrooney] I just finished a paper and someone said well did you think about
margin of error for these sample data and I said no and they said well maybe you shoult I could go back and research that at the census tract level instead of block level [Wray] it’s not really something I’ve suffered say you want to be able to present something and then maybe you’re not really allowed to [Mulrooney] these are the topic categories I think
there’s 21 of these topic categories we have an explanation of those in the
back of the standard here spatially referenced information you can see the
a difference between the ISO and the CSD GM so these are pretty self-explanatory
a lot this [Wray] this is also really critical if you’re just using GIS and it’s really
easy you just want to throw everything in your map and see what’s going on and
then something’s showing up here you know you gotta do some reprojection and
all that kind of stuff chances are also somebody may not have projected their
stuff correctly all of that kind of stuff so that’s why not only is that
part of the standard it’s really important to know where you’re dealing with what you’re dealing with [Mulrooney] metadata scope I
think this is first services so we have model feature attribute initiative for
service scopes so you can put service type and we’re developing the code list for service types and then these are just some examples of some optional elements
like the purpose the purpose comes up on the item description which is five
element metadata that I like to do with everything else and the currentness reference
which is always good because someone’s gonna ask you how current are the data and you don’t want to say I don’t know logical consistency in
this process date these are also very important and this was really important
to my dissertation because there’s some of the quantitative elements attached to
data quality you know these process state horizontal accuracy so when we look at some of these quality measures we actually go and create GIS data that’s what
we’re doing with my graduate class now is we’re evolving from me just giving
you a bunch of data to you actually going out creating a brand new data at
some point in time some new jerk in the back of the room that’s usually me in the class saying where to get these data from who created these
once you create these how do I know these are any good
so it’s pertinent that people you know populate these elements here
because I’m big into data quality before Sarah’s project we had we had a project
with the NC DOT we assessed data quality for all roads all 455,000 roads for the
state so we said in your this county and the best this county had the worst I had
a whole bunch of students working on it and they didn’t look at all
455,000 so we did some sampling procedures where we said you know the
counties between Mecklenburg and Guilford County were the best counties
kind of on the edges of North Carolina were the worst
we looked at data completeness we developed Python scripts to go through and
look at all these attributes whatever take and we’d still be
doing it for three or four years to look at this data quality so the main data
quality is really important especially small lessons I learned working for an
environmental agency and the military because these days for a lot of sources
that are spatialized you can turn tables tins relationship class
anything that you know just implies spaciality we can turn that into spatial
formats important important that that process is documented
these are all things I talk about in my classes here this is the NCC campus we just
bought a new toy drone so we’re able to you know take pictures we’re figuring
out how to use it kids in my class are so much they fly like video games
I’m really scared to kind of fly it around these kids can have it following them on
motorcycles and other things here so but this is a drone so it wouldn’t be
capturing this imagery or starting out this is just you know red green blue
we do have you do have an infrared camera on one of them on one of the
other drones that we bought you can see this from the NOAA website and we just
convert you know latitude and longitude values from the Climate Prediction
Center convert these project these and ing geographic coordinate system this is a
little thermal camera that I bought last year just attached to a phone so we’re
trying to figure out how to mount that on the
drone and take pictures there so it did for me it’s important that we’ve got
spatial information about all of these so these are also encapsulated with
the metadata and part of my dissertation was I started to pull all
these individual elements out of metadata some look at the horizontal
accuracy they’re talking about nine or ten quantitative elements compute this
is the process date just the vertical accuracy the horizontal accuracy how
many processing steps there are the process dates the currentness data so you
know basically I took all of these and tried to process information from and
tell my boss hey we’ve got 100 layers here even the ten oldest layers use the
ten layers that don’t have logical consistency so we can address them give
me a data really point-and-click or take a graduate student to point-and-click
you look for this information and I’ll talk about that a little bit there but
this is just some of the data quality information we have the contact and some of the contact information that we have here where’s the process description I went and GPS’d this
particular one in you know 2007 on these particular dates [Wray] That reminds me of one of the stories that … always tells not only you
documenting this for the people that are using your data but you’re documenting
for yourself your future self will thank you when you want to pull something back
up and then you’re like what did I what did I do with this and you actually have
the information about what you did with it with the data itself inherently
attached to it that’s also another big win for doing that documentation as part of the data creation itself and the data maintenance for that matter [Mulrooney] that’s why I encourage all my students
good as a GIS tech they’re not going to be doing a lot of analysis they’re doing
a lot of data development making the actual data you know whether it’s
parcels at some point in time you had these big flat maps that I work with
fifteen years ago now they’re all been digitized you’ll be updating those or
obtaining attributes go to the Beaufort County website that has a you know Steve
who’s on the metadata committee and within a couple of days of someone
selling a house that attibute information’s been updated they’re gonna
be doing things like that instead of you know doing the fancy analysis fire routes and
all the other things so I’m really to me I’m pretty passionate about data quality
creating data and we’re going to be adding to the body of knowledge that
NC OneMap can provide for everyone through their downloadable data and their services had a couple discussion questions here some of the metadata practices what are some of the
key or anyone who looks with metadata what is the best metadata practice by your
organization [Wray] is anybody writing metadata now [Audience member] as needed [Wray]
you know that’s uh we were talking with somebody two weeks ago and they work
private sector and they only write it if it’s paid for within the contract and
people are forgetting to put money and time in the contracts for the
deliverables for private sector and public sector or between each other to
actually get the data documented so that’s also an as needed as paid for in some cases as well and then we’re left
with high value but you know it degrades very quickly
beautiful beautiful I don’t know all sorts of planning information and then nothing to go along with it [Mulrooney] we have a
student working on a food desert project and he’s graduating next month starting
with ESRI you know about less than two months down in Charlotte had an
internship and I’m trying to get him to populate all the metadata cuz when he goes you
know and someone picks up on the project I don’t want to you know spend a
month going through it so we’re working feverishly on the metadata and I’m holding signing his thesis because it’s going to be difficult on the next group of folks there [Wray] and at NCDOT we don’t have an data governance rule put in places for metadata
right now but that’s what we’d be working toward so if somebody is going to right
now we have business units that publish their data they can publish for example like
crashes are published from that business unit it doesn’t come to the GIS unit but we don’t
actually have a data governance rule in place that says your metadata must
need the state profile so what we are looking to do is put more of that
governance in place so that if something is out there it at least has a standard
documented look coming from the agency so it’s another way to kind of enforce it try not to be the data police
I’m not trying to prevent information from getting out there but things as
important to the public is where the fatal crashes are you should actually
know how current that data is where it came from
is it the authoritative source or is it just somebody who’s playing around
because they all of a sudden got the permission to start publishing something
so from the state government perspective we want to make sure that it’s
understood authoritative sources and that kind of stuff so you have to weigh
that with getting them getting the documentation and people to care enough
to do the documentation to begin with versus the governance
in place and so for us that’s getting the profile in place and then
understanding how we’ll be able to apply that governance back out and that’s
basically the case with every federal agency we met with at the workshop that
Tim and I were at it was the alphabet soup of federal agencies and everybody
was suffering the same thing how to make people do it without driving them crazy
but still getting it done and even as late as this week the
profiles are continuing to move forward and they’re still struggling with implementation [Mulrooney]the biggest
challenge is the time and energy to populate it so when I worked with the
military if I do I’m going to be the process contact I’m going to be
the metadata contact I’m going to do all these other it’s going to be standard I
created some script that just auto-populated metadata so I’m looking
at ways and Python to do that you know it’s a little tricky with the
translators and whatnot and there’s multiple translators going from and to
the metadata but that would make life a lot easier I’ll show a couple
things that work we’re starting on right now and we go in the other direction
where you know particular values and it’s gonna make your life a lot easier [Wray] and the idea
of this committee and as Tim is going to show you is that we want to make it not
so mysterious like the videos that he shows this profile itself … it feels thick but really it’s just a table there’s not that many
things in there so to make it as easy as possible to understand what it is you’re
supposed to do and how to interact with it and some of the stuff that helps show
you coming up is how we’re trying to make it easier to do the work
so it doesn’t feel so overwhelming for the people [Audience member] one the best practices
in the past you used to get metadata and it would be in numerous different formats XML
HTML and then even some in Word docs and I guess you know obviously Word
docs I guess they’re easier for a vast majority to open but XML is most
useful especially if you’re going to use it within software you know be able to
open it in Arc Catalog and stuff like that so I guess one best
practice would need to be multiple formats or at minimum you know XML
format it can be converted [Mulrooney] I like XML personally but North Carolina OneMap also has metadata kits … why a profile it was developed to
address the core requirements of operational metadata and I think we have
a number of the street center lines multiple municipal boundaries and parcel
we have those on the North Carolina One Map and I’ll show some examples of those in a second also data custom content web characters finding a place or
county name the contact information sample abstracts sample
using strings that can use in there already you only need to change the
information that’s going to be pertinent to you [Wray] what we were finding is that there’s a
lot of sort of governmental based but there’s a lot of data that comes from
the local sources and then gets conflated as it moves up in the
governmental systems for local planning organizations the districts to state level to federal level that kind of stuff so what we
wanted to do is to look at these framework data sets of GBC framework
data sets and say is there something unique about that data set that we can
then say well here’s a template for how to fill out the template so that it
would be as easy as possible and maybe even standardized a little bit so
everybody that’s looking at that um that statewide parcel data set for example
that’s on NC OneMap if everybody starts to use the same constraint
information then you can kind of help we’re trying to help the local
governments pre-populate as much as possible so that they’re really just
worried about the intricacies of the things they need to update every time they get updated so that’s why we’ve got these different
framework templates [Vogler] yeah I just wanted to comment on the disclaimer statement we have a
group here at the center that uses a land change model called FUTURES and the
outputs from that are these land change prediction maps out to 2030 or 2040 and
whenever we share these with other researchers whoever requests we
definitely include a disclaimer with these predictive maps that basically
saying they’re landscape trends and not site specific outcomes please use caution when
using this with your own research so it is something we do think about [Wray] we want to make sure that we’re not we want to be editor agnostic so if
somebody is using open source we want to make sure that the XML template can be pulled into your open source software into your ArcGIS if you’re using ESRI software the CATMDEdit is one of
my favorites that’s open source it does really well with the template and it
provides better visual feedback I feel the QGIS editor has some gaps but it’s easy enough to
use and we’ll be looking for work with Boundless to kind of shore that up so
that it gives more of a visual feedback to some of our profile needs ESRI had a metadata summit out at
Redlands this summer and they took to heart a lot of what this metadata bootcamp that Tim and I attended
provided them probably more feedback from they were willing to accept at the time
with their particular editor and they’ve been able to convince their developers
to implement the metadata editing changes in their actual roadmaps for
the rollout of Arc Pro so things like AGOL right now you can publish the
service metadata but you can’t reference any of the layers within your AGOL map and so
somebody you just know the metadata about the service but nothing about
what’s inside of it that’s a big problem for AGOL implementations that is on now on the
ESRI roadmap for release next spring so these companies both open source of
course they’re taking the feedback but ESRI is also taking feedback to kind of
make metadata more accessible or easy to interact with to write and actually the
presence of it in some case to be strengthened [Audience member] Sarah, you mentioned on the
last slide frameworks layers parcels as an example what are some of the other
framework layers because some of us are probably using some of those [Wray] so cadastral
transportation so these are the also the ISO theme keywords
transportation boundaries tends to be one or particular for us we’d
wanted to say municipal boundaries because that’s a dataset that we had ortho imagery so this was kind
of our breakdown of some of the known statewide conflated data sets that we wanted to show [Mulrooney] couple things you know just what’s gonna
make and we’re always open to input about what’s gonna make metadata more
digestible you know facilitate the shift to ISO and what best practices are helpful [Wray] for you guys you don’t have to shift because you didn’t have it
to begin with so no matter if you didn’t have CSD GM there’s no migration so just
start with the ISO start with the the text XML templates you just load them in and go from there [Mulrooney] other things you know what about the profile
doesn’t work we haven’t had too much pushback on it [Wray] they just want us to come and tell them
more about it [Mulrooney] metadata next steps importing applying metadata profile to
a particular data set how to train your organization so that’s what we’re
going to be you know working on proposing talk about that in a second the assessment
so meta of metadata who’s completing at what extent and then
programmatically assess a profile compliance so some of the things I working on just a little Python tool we have data set output file XML folder and
we’re starting to starting to write metadata within a database and
write all the metadata entries to a single place here so this is ten years
ago in VBA and now finding the right in my free time but these
are some of the things that we can do and then if any of these are
quantitative in nature we can you know maybe not summarize these but we can
we can look at a number and whether it’s in meters or dates … and run some statistics on this information that’s kind of my next step my other
step looking is just turning this standard into you know quantitative
information so is the you know is the abstract populated yes you’re now
flag it so we’re going to your database we can say you know layer one
has you know fifteen flags on it these are the particular flags you know layer two
has two flags on it and you know it’s missing you five of the optional
elements and that’s just gonna shouldn’t be too hard programmatically it’s just
la lot of it’s resources intensive for all the permutations [Wray] and it’s missing out of a lot of editing
tools there there’s a ton of stuff on github from the feds there’s a NASA one
there’s one called GeoPlatform there’s there’s all sorts of stuff out
there the problem is the validation so yes we have a profile how do you know
you filled out a profile correctly well the problem with that is there wasn’t an approved
standard for the schema to map it against to validate that just was
approved and endorsed so these the companies now and the editors should be
able to start to implement the in-place validators yes in Arc Catalog there’s a
validate button but all that data is produced an exportable thing that you
have to do something with so and then in Arc Catalog there’s some other
things that if you set it up correctly it’ll think that something’s mandatory
but it’s not particular to this profile so we would want to do is take now that
there’s a schema to validate against maybe a just act from the North Carolina
profile and build that into an available validator similar to the old TSD GM
validator that USGS used to have metadata parser and if we’re lucky at all maybe they’ll update
it for the ISO standard and continue to use some of those validation
tools but I think this in working with Central and maybe you guys have some
ideas to get that validator certainly the next step is you spent all that time
or maybe not all that much time what else do I have to fill in what’s missing
or did you type in your date wrong you know whatever to do that validation [Audience member] so
you think they’ll get a chew and spit version too [Wray] I’m hoping I mean the
EPA decided not to wait around and wrote an add-in to Arc Catalog for it it’s downloadable it’s free it’s
tricky though because for example the contact list you can have an off site
reference contact list that you link to well it’s automatically built into their
tools so you have the entire EPA contact list if you go to use it so you have to
kind of go and change a couple of things unless you want to make EPA your contact lis but so there’s there’s some things that
we’re seeing out there that people were doing anyway what we want to do is
hopefully yeah chew and spit it out or ideally as you’re writing it it tells
you where to go next so you’re not having to go in and out that was the
pain in the butt with the USGS metadata parser was I had to export right they
the parser you see what wrong you got to go back to get back but there were
hundreds the tabs of the tabs of the tabs and the old stuff in and find your
way hopefully these tools that you’re interacting with will give you visual
feedback as you’re doing that entry what’s missing so far is the hope so you’re not
actually having to feed anything it’s all automatically there but if the vendors don’t come up with
that stuff the state is looking to see what we can do to help facilitate
getting something out there at least for us to validate … [Mulrooney] future training we’ve got some fliers up here but we’re looking to offer training and December 12 is usually when the semester lets out on our campus December 12 so contact me if you’re
interested or maybe send some students we’ll talk about fundamentals
and metadata components very intro level and we’ll talk a little bit about a data
quality and basic metadata assessment get real hands-on with ArcGIS and other software… how
to get hands on how to edit import export metadata and profiles that are
downloadable from North Carolina One Map that have been made available so you can contact me and I think we have labs with 16 students each so when they fill up and then one so we get
our dates for the spring based on availability of our lab I’ll post those dates and pass those along to Sara I’ll probably be teaching I’ve got graduate student and you know
once he goes away it should be pretty easy to keep training [Wray[ and it’s an
all-day training right [Mulrooney] yes all day [Wray] we could come here if somebody wanted us to do a lab
so we could come here if there’s a lab available and do that hands-on session doesn’t
have to be the whole day but we could do a hands-on lab as well [Mulrooney]we’ve got YouTube page so this is our YouTube page I’ve got a bunch of resources for our campus
you know the classes that I teach research presentations some of things
we’ve done with our drone but we’ve got a metadata training so these are some
of the metadata training tutorials that I posted some of them are pretty basic
a couple of them talk about the XML metadata and translators some get
into a little tool that I’ve created and bells and whistles in the tool how to view metadata how to use the standards so what we went through today and how to look up some of the basic entries on the standards So I’m probably going to add about four or five more of these [Wra] If you guys have questions please
feel free to put your name and email [Mulrooney] I’ve got the card up here and color profiles
and [Wray] we can send you all the links [Mulrooney] if you want a flier or anything feel free to take it and put my contact information there [Audience member] you know of any state agencies that have gone full bore into trying to implement
[Wray] I would think the EQ is the closest I know that DPS
so public safety with emergency management is certainly pays for a portion of any LIDAR collection so I know that stuff is mandated in there I
don’t know about their real time stuff but I don’t think anybody’s jumped in
the deep end just yet we are also still waiting on NC One Map to be ISO compliant as the local governments manage state agencies and any other data that they harvest starts to create it in the new profile they going to need to harvest it [Audience member] there were several counties that wanted to they don’t have any metadata and so
they were [Wray] yeah Guilford in particular was pretty lacking and that’s pretty all
sorts of different governmental reasons for that but yes they were among the folks that did not
even have to worry about doing any migration they could just start in ISO At NC DOT we’re about to jump in our deep end because I was told that’s what our newest group Chelsea and I would be doing in the governements and enforcing that came out of regional planning organization requests as well as some executives who want to make sure
that the engineer should know what data they’re working with and that we’re not double-paying for data collection and that kind of stuff so there’s a there’s a
political and a financial reason that we’re being asked to enforce very quickly that’s how that happens I think it’s correct I like it
I don’t have a problem with that assignment I’ll buy Chelsea a deputy badge you keep calling us the police I don’t think that’s right yeah she’s cool with it [Mulrooney] North Carolina One Maps metadata
resources so we’ve got the ISO metadata standards feature web services and these are some of the templates [Wray] yeah scroll up real quick so the PDFs for the ArcGIS and the cat
M are step-by-step instructions load the text file in here
go here fill out these things it’s literally click here do this from the
templates that are provided that’s right so the idea is to make them more interactive the trick with ESRI stuff is now that they’re releasing more often you don’t know how it’s going to change the idea is to keep this
website up to date the committee one of the committee’s jobs is to keep this
website up to date I know the cat MDA I haven’t looked at it
since February for the conference so we need to go back and revisit it or if you guys get a wild hair and you
decide you know I’m going to take a look at it and you want to send in an
update we’d be happy to take that update I said we’re not too proud we’ll take it
we’ll take anything we can get that helps the greater community and then we’ll add you to the committee if you want to be part of the committee let me know we are always
looking or any if you’re interested in any state level committee
participation let me know because they’re always looking they’re getting ready to stand up a classified land
use land cover Standards Committee for example so always looking for volunteers [Vogler] let’s thank Tim Sarah and Chelsea for
being here I thank you all for being here too and please after any questions
are asked and answered get some snacks on the way out

Daniel Ostrander

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *