* This transcript was created by voice-to-text technology. The transcript has not been edited for errors or omissions, it is for reference only and is not the official minutes of the meeting. [00:00:08] GOOD AFTERNOON. WE ARE CALLING TO ORDER THE TECHNOLOGY AND INNOVATION COMMISSION MEETING WEDNESDAY, DECEMBER 27TH, 2023. WE'LL BEGIN WITH OUR ROLL CALL. HI, [CALL TO ORDER] CHAIR LISA MAY BRUNSON. COMMISSIONER NICOLE CALLAHAN. COMMISSIONER JONATHAN DIAZ. YES. COMMISSIONER [ROLL CALL] CATHERINE KELLY. HERE. COMMISSIONER JAVIER ORTIZ. COMMISSIONER EMILY WOODMAN NANCE HERE. THANK YOU. WE HAVE A QUORUM. ITEM NUMBER ONE RECOMMENDATION TO APPROVE THE MINUTES FOR THE TECHNOLOGY AND INNOVATION COMMITTEE. SPECIAL MEETING OF WEDNESDAY, JULY 26. 2023. DO WE HAVE ANY PUBLIC COMMENT FOR ITEM NUMBER ONE [1. 23-51624 Recommendation to approve the minutes for the Technology and Innovation Committee special meeting of Wednesday, July 26, 2023.       Suggested Action: Approve recommendation. ] C? NONE. THANK YOU. MOTION CARRIES. ITEM NUMBER TWO, RECOMMENDATION TO RECEIVE AND FILE DIRECTOR'S REPORT. ALL RIGHT. UH, GOOD AFTERNOON, UH, COMMISSIONERS. UM, I KNOW IT'S BEEN A WHILE SINCE WE'VE HAD, YOU KNOW, A REGULAR MEETING, BUT A LOT HAS HAPPENED, UM, SINCE THEN. UH, FIRST WE DID HAVE THE RESIGNATION OF VICE CHAIR JUSTIN HECS, SO YOU'LL SEE LATER ON THE AGENDA, WE DO HAVE THE APPOINTMENT OF A NEW VICE CHAIR. WE, I KNOW THERE'S ONLY FOUR OF YOU HERE TODAY, SO WE COULD [2. 23-51625 Recommendation to receive and file Director’s Report.       Suggested Action: Approve recommendation. ] DISCUSS WHETHER WE MOVE FORWARD WITH THAT TODAY OR HOLD IT OFF FOR A MONTH. BUT WE CAN TALK ABOUT THAT WHEN WE GET TO THAT ITEM. UM, ALSO, WE HAVE A NEW COMMISSIONER JOINING US TODAY, OUR FIRST MEETING, AND THAT'S EMILY WOODMAN NANCE. AND ACTUALLY, EMILY, WOULD YOU LIKE TO JUST BRIEFLY INTRODUCE YOURSELF TO THE COMMISSION? YES, I'M AS STATED. I'M EMILY WHITMAN NANCE. UM, I HAVE DECADES OF TECHNOLOGY AND INNOVATION EXPERIENCE, AND I'M CURRENTLY THE CEO AND CO-FOUNDER OF AT THEIR SIDE, WHICH IS A DIGITAL HEALTHCARE TECH COMPANY. SO I'M JUST HAPPY TO BE HERE. AND I'M FROM DISTRICT FIVE TOO. WELL, WE'RE EXCITED FOR YOU TO JOIN OUR COMMISSION. UM, I DON'T KNOW IF THE COMMISSIONERS WANTED TO INTRODUCE THEMSELVES OR DID YOU ALREADY DO THAT BEFORE THE MEETING? SURE. MY NAME IS, COULD BE WATCHING. YEAH. HI, NICE TO MEET YOU. CATHERINE KELLY. AND I'M A, UM, MARKETER AT A TECH COMPANY, AND I LIVE IN DISTRICT TWO. NICE TO MEET YOU. UM, UH, JONATHAN DEIS. AND, UH, I AM A FOUNDER OF ISLAND PITCH, UM, IT HERE IN LONG BEACH. IT'S A INNOVATION CONSULTING COMPANY. AND, UM, I HAVE BEEN ON THE COMMISSION NOW FOR A LITTLE OVER A YEAR. UM, WELCOME, WELCOME, EMILY. I'M EXCITED TO HAVE YOU ON THE COMMISSION. UH, I'M CHAIR OF THE COMMISSION. LISA MA B BRUNSON. WE KNOW EACH OTHER IN THE INDUSTRY AND IN THE COMMUNITY. I THINK I'M A DISTRICT TWO. UH, ARE WE TWO? WE, I'M THREE. OH, OKAY. SO TWO. HE'S MY NEIGHBOR SOMEWHERE IN THE AREA. BUT ANYWAY, WELCOME TO THE COMMISSION. I HOPE THAT WE DO SOME GREAT WORK TOGETHER IN THE NEW YEAR. OKAY. YEAH. AND, UM, THERE'S A COUPLE OF NOTEWORTHY THINGS THAT HAPPEN ALSO WITH THE CITY AND TID THAT I WANTED TO ANNOUNCE. UM, ONE WAS FOR THE THIRD CONSECUTIVE YEAR, WE EARNED THE FIRST PLACE DIGITAL CITY AWARD, UM, BY THE CENTER FOR DIGITAL GOVERNMENT. AND THAT'S UNHEARD OF TO GET THIRD FIRST PLACE THREE YEARS IN A ROW. SO IN, IN OUR POPULATION CATEGORY. AND WE'VE HAD, UM, THE BE WE WERE IN THE TOP 10 FOR 13 YEARS IN A ROW. SO THAT'S A SIGNIFICANT ACCOMPLISHMENT. AND ALSO WE WERE, [00:05:01] WE RECENTLY, JUST THIS MONTH WE'RE HONORED AS A VISIONARY DIGITAL INCLUSION TRAILBLAZER, UM, BY THE NATIONAL DIGITAL INCLUSION ALLIANCE. AND WE'VE RECEIVED THE DIGITAL INCLUSION TRAILBLAZER AWARD FOR SIX YEARS. ONE OF 47 JURISDICTIONS IN THE NATION, BUT ONLY ONE OF 22 WHO GOT THE TOP DESIGNATION OF THE VISIONARY DIGITAL INCLUSION TRAILBLAZER. SO THAT, THAT'S ANOTHER NOTEWORTHY ACCOMPLISHMENT. UM, AND THEN YOU PROBABLY ALL HEARD WE DID HAVE A, A NETWORK CYBERSECURITY INCIDENT IN NOVEMBER, AND THERE'S NOT MUCH I CAN SAY ABOUT THAT BESIDES WHAT WAS ALREADY PUBLICLY REPORTED, BUT THAT HAS DEFINITELY BEEN A VERY SIGNIFICANT, UM, INCIDENT FOR THE CITY AND THE TID TEAM. AND WE, WE ARE CONTINUING TO WORK ON THE INVESTIGATION, BUT OUR SYSTEMS ARE NOW BACK UP AND RESTORED AND FUNCTIONING PROPERLY, WHICH WAS A, A BIG ACCOMPLISHMENT IN TERMS OF JUST THE AGENDA BEFORE YOU, UM, THE, WHILE THIS COMMISSION AS A BODY HASN'T MET REGULARLY, THE GOOD NEWS IS WE HAD OUR AD HOC COMMITTEE MEETINGS, UH, REGULARLY. SO YOU'LL BE HEARING MORE ABOUT THAT UNDER ITEM FOUR. UM, AND WE ALSO, UH, ARE READY TO PRESENT A CIVIC USER TESTING PLAYBOOK FOR ITEM THREE. AND YOU LEARN MORE ABOUT THAT IN THAT ITEM. AND THEN WE WILL, UM, AS A PART OF ITEM FOUR, IN ADDITION TO HEARING ABOUT WHAT THE GENERATIVE AI UH, COMMISSION'S UP TO, WE CAN TALK ABOUT OTHER POTENTIAL AD HOC COMMITTEES AND OTHER THINGS. AND THEN INFORMALLY IN, IN ITEM FIVE, WE'RE GONNA ASK YOU TO ESTABLISH THE DATA PRIVACY AD HOC COMMITTEE. UM, AND THEN IF WE ARE READY, WE CAN HOLD THE ELECTION FOR THE VICE CHAIR. AND THAT'S TODAY'S AGENDA. AND WITH THAT, THAT CONCLUDES MY REMARKS. THANK YOU, LEAH. SO NOW ARE WE GOING TO, UH, WE'VE ALREADY MOVED IN. DO SECONDED. DO WE HAVE ANY PUBLIC COMMENT FOR ITEM NUMBER TWO? SEE NONE. COMMISSIONER KELLY, IF YOU CAN. THANK YOU. MOTION CARRIES. ITEM NUMBER THREE, RECOMMENDATION TO RECEIVE AND FILE A REPORT ON THE DRAFT CIVIC USER TESTING PLAYBOOK. RIGHT. FOR THIS ITEM, I'M GONNA TURN IT OVER TO RYAN KURTZMAN, WHO IS THE TECHNOLOGY PARTNERSHIPS OFFICER FOR THE TECHNOLOGY AND INNOVATION DEPARTMENT. GREAT, THANK YOU, LEAH. AND I'M ACTUALLY GONNA TURN IT RIGHT OVER TO AMI LI, WHO IS A MANAGEMENT ASSISTANT WHO WILL BE PROVIDING A REPORT [3. 23-51626 Recommendation to receive and file a report on the draft Civic User Testing Playbook.       Suggested Action: Approve recommendation. ] VIRTUALLY ON THIS ITEM. THANK YOU, RYAN. UM, I JUST WANNA MAKE SURE, CAN EVERYBODY SEE THE SCREEN IN THE PRESENTATION? YES, WE CAN. THANK YOU. AWESOME. WELL, HI EVERYBODY. UH, GOOD AFTERNOON. MY NAME IS AMI LI. UM, I AM A MANAGEMENT ASSISTANT WITH THE CITY OF LONG BEACH. AND, UM, I HAVE THE PLEASURE OF PRESENTING, UH, MY WORK THAT I DID ON THE CIVIC USER TESTING PLAYBOOK WHEN I WAS ROTATING THROUGH THE TECHNOLOGY AND INNOVATION DEPARTMENT. SO, JUST TO GO OVER THE AGENDA OF WHAT I'LL BE COVERING IN THIS PRESENTATION, UM, I'LL FIRST GO OVER THE PROJECT OVERVIEW, UM, FOR CREATING A, UH, CIVIC USER TESTING PLAYBOOK. UM, THEN I'LL JUMP INTO THE METHODOLOGY THAT I USE TO CREATE THE FIRST DRAFT OF THE PLAYBOOK. UM, THEN I'LL GO OVER HOW I COLLECTED FEEDBACK, UM, FROM THIS FIRST DRAFT. AND THEN, UH, WE'LL DIVE INTO, I'LL PRESENT THE OUTLINE OF THE PLAYBOOK, UM, JUST FOR TIME'S SAKE, I'LL GIVE A GENERAL, UH, OUTLINE OF THE PLAYBOOK. I WON'T GO INTO TOO MUCH DETAIL, UM, BECAUSE IT, IT IS A, A, THERE IS, THERE ARE EIGHT SECTIONS TO THE PLAYBOOK. AND THEN I'LL GO OVER HOW THE PLAYBOOK IS MEANT TO BE USED. AND THEN, UH, FINISH UP WITH NEXT STEPS FOR, UH, IMPLEMENTING THE CIVIC USER TESTING PLAYBOOK. SO I'LL DIVE RIGHT INTO IT. UM, SO FIRST, A PROJECT OVERVIEW. SO BEFORE WE BEGIN, I JUST WANNA MAKE SURE THAT WE ARE ALL, UM, HAVE THE SAME UNDERSTANDING OF THE DEFINITION OF WHAT CIVIC USER TESTING IS. UM, SO CIVIC USER TESTING, SOMETIMES YOU'LL HEAR ME SAY CUT AS AN ABBREVIATION, BUT IT IS A PROCESS TO USE TO GATHER PUBLIC FEEDBACK TO EVALUATE THE USABILITY AND EFFECTIVENESS OF DIGITAL SERVICES AND PRODUCTS INTENDED FOR PUBLIC USE THROUGH ACTIVE COMMUNITY OUTREACH AND ENGAGEMENT. UM, SO IN THE PRIVATE SECTOR, IT'S USUALLY KNOWN AS USER TESTING OR USABILITY TESTING. BUT, UM, ONE DISTINCTION WITH CIVIC USER TESTING IS IT'S MORE INTENTIONAL IN INVOLVING THE PUBLIC IN CREATING, UH, A SPACE TO CO-CREATE AND DESIGN, UM, DIGITAL SOLUTIONS TOGETHER. UM, IT IS, IT'S WITH THIS BUILD WITH MENTALITY INSTEAD OF THIS BUILD FOR APPROACH. UM, AND ESSENTIALLY IT INVOLVES, UH, RECRUITING A DIVERSE SET OF PARTICIPANTS [00:10:01] THAT REPRESENT THE TARGET AUDIENCE OF THE DIGITAL SOLUTION OR PRODUCT. UM, HAVING THOSE, UH, PARTICIPANTS THEN INTERACT, UH, WITH THE DIGITAL SERVICE, PERFORM A SERIES OF TASKS, UM, AND HAVE STAFF, UH, TAKE NOTES AND COLLECT FEEDBACK ON THE OBSERVATIONS AND CHALLENGES THAT, UH, PARTICIPANTS FACED WHEN, UH, UTILIZING THE DIGITAL TOOL. UM, AND THEN TAKING ALL THAT FEEDBACK AND THEN, UH, MAKING THAT PRODUCT MORE ACCESSIBLE, UH, AND USER FRIENDLY FOR THE PUBLIC. UM, SO MY PROJECT WAS TO DEVELOP A CIVIC USER TESTING PLAYBOOK FOR CITY STAFF. THAT INCLUDES BEST PRACTICES, STANDARDS, TEMPLATES, GUIDES, TRAINING, AND ADDITIONAL MATERIALS TO FACILITATE THE USE OF CIVIC USER TESTING AND THE DEVELOPMENT OF DIGITAL SERVICES. UM, DEVELOPMENT OF THE PLAYBOOK USED AN EQUITY LENS TO PROMOTE ACCESSIBILITY, TRANSPARENCY, AND PUBLIC COMMUNICATION FOR THE CITY'S DIGITAL SERVICES. UM, SO THIS, UH, PLAYBOOK IS SIGNIFICANT, UM, BECAUSE IT IS, UH, IN ALIGNMENT WITH THE SMART CITY INITIATIVE STRATEGY AND 2030 STRATEGIC VISION, UM, WHERE THE CITY IS COMMITTED TO DEVELOP, UH, CHANNELS FOR COMMUNITY CO-DESIGN WITH RESIDENTS, SMALL BUSINESSES AND OTHER STAKEHOLDER GROUPS TO CENTER USER EXPERIENCE TO THE END USER IN THE DEVELOPMENT OF DIGITAL SERVICES. SO BY INCORPORATING USER FEEDBACK AND FOCUSING ON USER-CENTERED DESIGN, GOVERNMENTS AND ORGANIZATIONS CAN ENHANCE THE OVERALL EXPERIENCE AND ENSURE THAT DIGITAL SERVICES ARE USER-FRIENDLY, ACCESSIBLE, INCLUSIVE, AND EFFECTIVELY MEET, UH, MEET THE NEEDS OF ALL INTENDED USERS. SO AT THE END OF THE DAY, THIS IS ANOTHER, UH, INITIATIVE TO PROMOTE EQUITY AND ACCESSIBILITY AND DIGITAL INCLUSION. UM, BECAUSE THE LAST THING WE WANT IS TO SPEND, UM, A LOT OF STAFF RESOURCES, UH, CITY RESOURCES, MONEY FUNDS TO CREATE AND DEVELOP A DIGITAL SOLUTION THAT IS NOT USER-FRIENDLY, THAT IS' INACCESSIBLE, AND THAT PEOPLE ARE NOT GONNA UTILIZE BECAUSE IT'S SO, UM, BECAUSE IT DIDN'T TAKE INTO ACCOUNT THE PERSPECTIVES AND THE NEEDS, UM, AND INVOLVE THE END USERS, WHICH IS THE PUBLIC. UM, SO THIS, AGAIN, CIVIC USER TESTING HELPS, UM, CR HELPS CENTER THE END USER, CENTER THE PUBLIC IN CREATING THESE DIGITAL SOLUTIONS AND IMPLEMENTING THESE DIGITAL SOLUTIONS, UM, TO CREATE PRODUCTS THAT ARE ACCESSIBLE FOR EVERYBODY. UH, SO THE METHODOLOGY THAT I USED, UM, THE FIRST THING THAT I DID WAS I REVIEWED THE REFERENCE MATERIALS PROVIDED BY THE SMART CITIES TEAM AND OTHER, UH, INDEPENDENT RESEARCH. UM, SO I INCLUDED OTHER ARTICLES, PAPERS, UH, PUBLISHED PAPERS, MEDIA BOOKS, UM, I ALSO LOOKED AT WHAT OTHER CITIES WERE DOING, CITIES IN THE EAST COAST, CITIES IN TEXAS, CITIES IN THE SOUTH, UM, THROUGHOUT CALIFORNIA AS WELL. AND SAW WHAT THEY WERE DOING IN REGARD TO USER EXPERIENCE, UH, CIVIC USER TESTING, CHECKED OUT THEIR WEBSITES. SOME OF THEM ALSO HAD TOOLKITS FOR USER EXPERIENCE, FOR CIVIC USER TESTING. SO I TOOK A LOOK AT ALL OF THAT TO TRY IDENTIFY BEST PRACTICES. UM, THE NEXT THING I DID WAS I CONDUCTED STAKEHOLDER INTERVIEWS. SO I DID CONDUCT 10 QUALITATIVE INTERVIEWS WITH CITY STAFF. UM, IT WAS CITY STAFF WHO HAVE EXPERIENCE MANAGING AND IMPLEMENTING, UM, PROJECTS, UH, FOR DIGITAL SERVICES AND PRODUCTS. UM, THEY INCLUDED STAFF FROM FINANCIAL MANAGEMENT FROM TID, FROM PUBLIC WORKS, FROM THE CITY MANAGER'S OFFICE. I ALSO SPOKE WITH, UM, STAFF FROM THE OFFICE OF EQUITY AND LIBRARY SERVICES. UM, AND I ALSO CONDUCTED INTERVIEWS WITH OUTSIDE AGENCIES, INCLUDING THE CITY OF ASHEVILLE AND THE CITY OF BOSTON. UM, THESE INTERVIEWS UNCOVERED MAJOR CHALLENGES TO IMPLEMENTING CIVIC USER TESTING WITHIN THE CITY OF LONG BEACH, AND ALSO HELPED IDENTIFY BEST PRACTICES THAT OTHER CITIES WERE, WERE DOING, UM, IN REGARDS TO CIVIC USER TESTING. UH, ONCE I CONDUCTED THOSE INTERVIEWS AND DID MY INITIAL RESEARCH, UM, I CONDUCTED FURTHER RESEARCH INFORMED BY THOSE INTERVIEW FINDINGS AND MY INITIAL INITIAL SCAN. UM, AND THEN I COMPILED ALL THE RESEARCH AND FINDINGS TO CREATE A FIRST DRAFT OF THE CIVIC USER TESTING PLAYBOOK. SO ONCE I HAD THE FIRST DRAFT CREATED, WHAT I DID IS I WANTED TO GET FEEDBACK. UM, 'CAUSE AT THE END OF THE DAY, I GUESS THE END USERS FOR THE CIVIC USER TESTING PLAYBOOK ARE CITY STAFF. UM, SO I CONDUCTED TWO FOCUS GROUP SESSIONS WITH CITY STAFF, UM, ACROSS DEPARTMENTS. UH, I USED MURAL BOARDS TO GUIDE GROUP DISCUSSION AND RECORD PARTICIPANTS' FEEDBACK. SO IT WAS A SET OF EIGHT QUESTIONS THAT I HAD IN THE FOCUS GROUPS. UM, THE DISCUSSIONS COVERED SEVERAL TOPICS SUCH AS GENERAL FIRST IMPRESSIONS OF THE FIRST DRAFT, THE CIVIC USER TESTING PLAYBOOK, WHAT WORKED WELL, AREAS FOR IMPROVEMENT AND FEEDBACK ON SPECIFIC TOOLS INCLUDED IN THE CIVIC USER TESTING PLAYBOOK, THAT'S SUCH AS EQUITY CHECKS, SAMPLE LANGUAGE TEMPLATES. AND I'LL DIVE INTO, UH, MORE, UH, INFORMATION ON THOSE TOOLS THAT ARE INCLUDED IN THE CIVIC USER, UH, PLAYBOOK LATER ON IN THE PRESENTATION. UM, SO ONCE I CONDUCTED THOSE FOCUS GROUP SESSIONS, I ANALYZED, UM, THE FEEDBACK AND IMPLEMENTED HIGH PRIORITY CHANGES INTO WORKING VERSION OF THE PLAYBOOK AND IDENTIFIED COMMON THEMES, CHALLENGES, AND ISSUES THAT PEOPLE HAD WITH THE PLAYBOOK. SO [00:15:01] NOW LET'S DIVE INTO THE OUTLINE OF THE PLAYBOOK. SO THERE ARE, THE PLAYBOOK IS MADE UP OF EIGHT MAJOR SECTIONS. UM, FIRST IT STARTS WITH INTRODUCTION, UM, AND THEN THERE ARE SIX MAJOR COMPONENTS OR PHASES OF CIVIC USER TESTING. SO EACH OF THOSE MAJOR PHASES, PHASES IS A ITS OWN SECTION WITHIN THE PLATE. SO WE HAVE GETTING STARTED AS THE FIR AS THE FIRST STEP. IT'S ITS OWN SECTION. THEN WE HAVE UX METHODS AND PROTOTYPING, RECRUITMENT, TEST DESIGN, TESTING SESSIONS, ITERATIVE DESIGN. AND THE LAST SECTION IS ADDITIONAL RESOURCES. SO IT PROVIDES, UM, LINKS TO WEBSITES TO, UH, OTHER TOOLKITS TO OTHER, UH, RESOURCES AND READING MATERIAL IF PEOPLE WANNA DO FURTHER RESEARCH ON DIFFERENT ASPECTS OR DIFFERENT COMPONENTS OF CIVIC USER TESTING. SO I'LL GO AHEAD AND JUST GIVE A, UM, A BRIEF OVERVIEW OF WHAT CONTENT IS CONTAINED WITHIN EACH SECTION. SO FIRST, STARTING OFF WITH THE INTRODUCTION, UM, I, WE GO INTO DEFINING, PROVIDING A DEFINITION, A COMMON DEFINITION FOR CIVIC USER TESTING, UH, EXPLAINING THE IMPORTANCE AND RELEVANCE OF CIVIC USER TESTING. UM, WHILE ELABORATING ON THE BENEFITS OF USING CUT AND THE RISKS OF NOT UTILIZING CUT, IT GOES OVER HOW, UH, CITY STAFF CAN USE THE PLAYBOOK. UH, IT PROVIDES EXAMPLES OF WHEN IT IS RELEVANT TO UTILIZE CIVIC USER TESTING. UM, BECAUSE AT THE END OF THE DAY, WHILE THIS PLAYBOOK DOES CONCENTRATE OR FOCUS ON, UM, UTILIZING CIVIC USER TESTING FOR DIGITAL SERVICES AND PRODUCTS, IT CAN BE USED FOR A WIDE VARIETY OF PROJECTS. FOR EXAMPLE, WHEN, UH, CITY STAFF ARE DEVELOPING AN APPLICATION TO APPLY FOR A CERTAIN PROGRAM, UM, SOMETIMES WE THINK THAT OUR APPLICATIONS MAKE SENSE, BUT AT THE END OF THE DAY, WHEN IT GOES DOWN TO THE APPLICANT, SOMETIMES THEY MAY NOT COMPLETELY UNDERSTAND WHAT WE'RE ASKING, WHAT INFORMATION WE'RE TRYING TO COLLECT. SPECIFIC USER TESTING ALSO HELPS IN THAT REGARD TO MAKE SURE THAT WE ARE MAKING SENSE WHEN WE ARE CREATING APPLICATIONS. UM, ALSO FOR, FOR EXAMPLE, FOR PUBLIC HEALTH CAMPAIGNS AS WELL. UM, ANY MATERIALS THAT WE DEVELOP, UM, TO GET A MESSAGE ACROSS, WE WANNA MAKE SURE THAT, UH, THE MESSAGE THAT WE'RE TRYING TO GET ACROSS IS WHAT PEOPLE ARE PERCEIVING AND NOT SOMETHING ELSE. SO, UH, CIVIC USER TESTING HAS APPLICABILITY FOR A VARIETY OF PROJECTS, NOT JUST AT DIGITAL SERVICES AND PRODUCTS. UM, SO JUST LIST SOME OF THOSE EXAMPLES AS WELL. AND WHEN, UH, ENGAGING IN CIVIC USER TESTING WOULD BE USEFUL. AND THEN IT JUST GIVES AN OVERVIEW OF THE MAJOR STEPS OF CIVIC USER TESTING AND THE PROCESS. UM, THEN IT GOES INTO THE FIRST STEP OF CIVIC USER TESTING, WHICH IS GETTING STARTED. SO PLANNING FOR CUT, UH, IT GOES OVER, UM, ADVICE AND RECOMMENDATIONS FOR INCLUDING CUT IN, UH, RFPS AND VENDOR CONTRACTS STARTING EARLY, UH, DESIGNING FOR EQUITY, RIGHT, UM, POSING QUESTIONS, UH, GETTING THE PROJECT MANAGERS TO START THINKING ABOUT HOW THEY'RE GOING TO APPROACH THIS, UH, SOLUTION, DIGITAL SOLUTION IN AN EQUITABLE WAY. UH, AND THEN ALSO BUDGETING ADEQUATE RESOURCES AND TIME FOR CUT FOR THE CUT PROCESS. 'CAUSE AT THE END OF THE DAY, MATERIALS HAVE TO GET, UH, TRANSLATED. WE HAVE TO MAKE SURE THAT WE ALLOW OURSELVES ENOUGH TIME TO CONDUCT CIVIC USER TESTING BEFORE WE LAUNCH OR IMPLEMENT A PROJECT, UH, A DIGITAL SERVICE CITYWIDE. UH, THE NEXT, UH, SECTION IS UX METHODS AND PROTOTYPING. UM, SO THIS SECTION JUST GOES OVER VARIOUS USER EXPERIENCE METHODS AND RESEARCH PRACTICES THAT PROMOTE COMMUNITY ENGAGEMENT. UH, ONE OF THE THINGS THAT I UNCOVERED AND BEST PRACTICES FROM THE CITY OF ASHEVILLE, FROM THE CITY OF BOSTON, THEY SAID THAT EVEN BEFORE THEY STARTED BUILDING THEIR, BEFORE THEY HAD A PROTOTYPE THAT THEY WERE ABLE TO TEST OUT ON THE PUBLIC, IT WAS ALWAYS IMPORTANT TO START INVOLVING THE PUBLIC EARLY ON DURING THE DESIGN AND BRAINSTORMING SESSIONS, JUST TO MAKE SURE THAT THE PROJECT, THE PROTOTYPE THAT IS BEING DEVELOPED IS CENTERED AROUND THE END USER, IS CENTERED AROUND, UM, THE, THE NEEDS OF THE PUBLIC. UM, SO IT, THE, THIS SECTION ALSO ENCOURAGES USER-CENTERED APPROACH TO DESIGN AND THE IMPORTANCE OF DEFINING END USERS AND PROVIDES, UM, SOME EXAMPLES OF ACTIVITIES THAT THE STAFF CAN DO TO, UH, DEFINE END USERS, SUCH AS USING PERSONAS OR STARTING DOING AN INTERVIEW OR SURVEYING, UM, THE TARGET AUDIENCE OF THE DIGITAL TOOL OR SERVICE. AND THEN THE NEXT SECTION IS RECRUITMENT FOR, UH, CIVIC USER TESTING. SO IT PROVIDES GUIDELINES TO CREATE EFFECTIVE AND INCLUSIVE COMMUNICATION MATERIALS. UM, HOW TO CREATE A SIGNUP FORM THAT COLLECTS ADEQUATE INFORMATION FOR RECRUITMENT, RIGHT? UH, WE WANT STAFF TO RECRUIT A DIVERSE REPRESENTATIVE, UH, GROUP OF PARTICIPANTS THAT, AGAIN, REPRESENT THE TARGET AUDIENCE FOR THE DIGITAL TOOL. AND, UM, DIFFERENT OUTREACH MESSAGE METHODS THAT CAN BE UTILIZED FOR RECRUITMENT, SUCH AS EMAIL CAMPAIGNS, SOCIAL MEDIA, LEVERAGING, UH, RELATIONSHIPS WITH, UH, COMMUNITY-BASED ORGANIZATIONS TO REACH OUT TO, UH, HEARTS REACH POPULATIONS. THEN THE NEXT SESSION IS TEST DESIGN. [00:20:01] SO IT PROVIDES INSTRUCTIONS TO DESIGN A USER TEST. UM, ALSO PROVIDES GUIDELINES TO CREATE TEST SCENARIOS AND TASKS. SO DEPENDING ON WHAT DIGITAL SOLUTION IS CREATED OR NEEDED, UH, WE WANNA COME UP WITH DIFFERENT TEST SCENARIOS THAT ARE GOING TO, UH, GAUGE THE EFFECTIVENESS, UH, OF THE, OF THE TOOL. UH, AND IF IT'S, IF IT'S BEING UTILIZED IN, IN A WAY THAT IS MEANT TO BE UTILIZED, IF THE PURPOSE IS COMING ACROSS, UH, THEN WE GO OVER VARIOUS, UH, TESTING METHODS. THERE IS A MODERATED TESTS VERSUS MODERATED TESTS. THERE ARE THOSE THAT ARE MORE QUALITATIVE VERSUS THOSE THAT ARE MORE QUANTITATIVE, UM, THAT ARE THOSE THAT MAY BE DONE IN PERSON OR REMOTE. UM, SO IT JUST PROVIDES A DIFFERENT TESTING METHODS THAT STAFF MAY UTILIZE. UM, IT EXPLAINS THE BENEFITS AND DETRIMENTS OF EACH ONE, UM, AND IT'S REALLY UP TO THE STAFF TO, UH, DETERMINE WHICH TESTING METHOD IS BEST UTILIZED OR APPLIES TO THEIR SPECIFIC PROJECT. UM, AND THEN LASTLY, WE INCLUDE SOME INSTRUCTIONS FOR HOW TO WRITE, CREATE A TEST PLAN. UM, THE NEXT SESSION IS ACTUALLY CONDUCTING THE TESTING SESSIONS. SO IT GOES OVER INFORMATION, HOW TO MODERATE A TESTING SESSION, HOW TO RUN ONE, UM, CREATING A TEST SCRIPT, WHICH HELPS ENSURE CONTINUITY ACROSS TESTS, UH, HOW TO RECORD AND COLLECT DATA DURING THOSE TESTING SESSIONS, UM, AND OTHER LOGISTICAL MATTERS SUCH AS PROVIDING FOOD AND SNACKS, AND ALSO COMPENSATING PARTICIPANTS ALWAYS COMPENSATING, UH, THE PUBLIC FOR THEIR TIME, FOR TAKING TIME FROM THEIR DAY TO COME AND, UH, TEST OUT THESE DIGITAL SERVICES FOR US. UH, AND ALSO SETTING EXPECTATIONS ON COMPENSATION. UH, NEXT WE HAVE ITERATIVE DESIGN. SO ONCE THE PROTOTYPE HAS BEEN MADE, UH, WE HAVE RECRUITED PARTICIPANTS, UH, CREATED OUR TESTS, OUR TESTS, TESTING SCENARIOS, ACTUALLY RUN THE TESTING SESSION. THEN IT'S TIME TO ANALYZE ALL THE DATA FROM THE TESTING SESSIONS, UH, DEVELOP INSIGHTS, AND THEN DEVELOP ACTIONABLE, ACTIONABLE RECOMMENDATIONS FROM THOSE INSIGHTS. SO, IDENTIFYING THE ROOT CAUSES OF RECURRING CHALLENGES THAT WE OBSERVED IN THE TESTING SESSIONS AND DEVELOPING, UH, WAYS TO ADDRESS THOSE. UM, THEN IT GOES OVER HOW TO CREATE A TEST PLAN. AGAIN, THAT JUST SUMMARIZES, UM, ALL THE RESEARCH THAT WE'VE DONE AND THE INSIGHTS FROM THE TESTING SESSIONS AND THE RECOMMENDATIONS, UM, SO THAT WE'RE ABLE TO SYNTHESIZE WHAT IT, OR ANALYZE WHAT IS FEASIBLE AND WHAT CHANGES CAN BE DONE IMPLEMENTED, DEPENDING ON A STAFF CAPACITY, RESOURCES, UH, UH, HOW MUCH FUNDS WE HAVE, ET CETERA. UM, AND THEN SOMETIMES IF, UH, TIME ALLOWS, UM, AND THERE'S ENOUGH STAFF CAPACITY, SOMETIMES, UH, IT IS BENEFICIAL TO CONDUCT A SECOND USER TEST. SO ONCE WE'VE DONE AN INITIAL ONE, UM, IMPLEMENTED SOME CHANGES, CONDUCTING ANOTHER ONE TO MAKE SURE THAT THOSE CHANGES WERE EFFECTIVE. AGAIN, IT ALL DEPENDS ON THE PROJECT TIMELINE AND IF THERE ARE ENOUGH RESOURCES TO DO SO. AND THEN FINALLY, THE LAST STEP, UH, OR THE LAST SECTION JUST PROVIDES ADDITIONAL INFORMATION ON RESOURCES. AGAIN, IF THERE ARE STAFF WHO WANT TO, UM, READ UP MORE ON DIFFERENT COMPONENTS OF CIVIC USER TESTING, THERE ARE LINKS TO PAPERS, TO DIFFERENT WEBSITES, WHAT OTHER CITIES ARE DOING. AND THEN IT ALSO PROVIDES A LINK TO CENTRALIZED TOOLS, TEMPLATES, AND GUIDES THAT ARE CONTAINED WITHIN THE PLAYBOOKS FOR EASIER ACCESS FOR STAFF. UM, SO HOW TO USE THE PLAYBOOK, UM, THE PLAYBOOK IS MODULAR, AND IT'S NOT MEANT TO REREAD IT IN ITS ENTIRETY, AS I SAID, IT, IT, THE MAJOR SECTIONS OF THE PLAYBOOK ARE DIFFERENT STEPS TO CIVIC USER TESTING. SO STAFF MAY USE A TABLE OF CONTENT, UH, CONTENTS TO IDENTIFY SECTIONS MOST RELEVANT TO THEM, DEPENDING ON THEIR KNOWLEDGE OF CIVIC USER TESTING. UM, MY STAFF WOULD LIKE MORE INFORMATION ON CERTAIN PHASES OF CUT, ADDITIONAL READING MATERIALS ARE PROVIDED IN EACH SECTION. FOR EXAMPLE, SOMEONE MAY BE WELL-VERSED IN COMMUNITY ENGAGEMENT AND REACHING OUT TO HEART TO REACH POPULATIONS THEY MAY NOT NEED, UH, TO SPEND MUCH TIME READING THAT SECTION, BUT THEY MAY NOT KNOW HOW TO CREATE A TEST TO PLAN OR, UH, CREATE TASKS SO THEY CAN TAKE WHAT THEY NEED FROM THE PLAYBOOK. AGAIN, IT'S NOT MEANT TO BE REDDITS ENTIRETY. UM, AND THE PLAYBOOK DOES CONTAIN VARIOUS TOOLS, GUIDES, TEMPLATES, INSTRUCTIONS, AND EXAMPLES FOR STAFF TO USE AS NEEDED. SO SOME OF THOSE TOOLS, UH, SO THERE ARE TEMPLATES. SO FOR EXAMPLE, THIS ONE THAT I, ON THE LEFT SIDE THAT I HAVE ON THE SCREEN, UM, IS A TEMPLATE FOR HOW TO, UH, RECRUIT. IT'S A RECRUITMENT EMAIL. UM, SO HOW TO REACH OUT TO, UH, COMMUNITY MEMBERS TO RECRUIT THEM TO BE, UH, CIVIC USER TESTERS. SO IT'S A TEMPLATE THAT, AGAIN, CITY STAFF CAN TAKE. UM, AND THEN JUST PLUG IN THE RELEVANT INFORMATION ON THERE. THERE'S SEVERAL TEMPLATES INCLUDED WITHIN THE, UH, CIVIC USER TEST AND PLAYBOOK. THERE ARE OTHER TOOLS, FOR EXAMPLE, THIS PROVIDES A GENERAL FORMAT OF A TEST SCRIPT. SO THE DIFFERENT COMPONENTS THAT SHOULD BE INCLUDED IN A TEST SCRIPT. SOME OTHER TOOLS WE HAVE, UM, THESE ARE CALLED EXAMPLES AND ACTIONS. SO IT'S A LOT OF TIMES IT'S EASIER TO UNDERSTAND CERTAIN CO [00:25:01] CONCEPTS, UM, DESCRIBED IN THE PLAYBOOK, UM, BY PROVIDING REAL LIFE EXAMPLES. SO I TOOK SOME TIME TO PROVIDE, UM, EXAMPLES WHERE, UH, THE CITY OF LONG BEACH HAS CONDUCTED CIVIC, CIVIC USER TESTING OR THE CITY OF BOSTON OR OTHER CITIES, UM, JUST AGAIN, TO BETTER, UH, HELP STAFF UNDERSTAND CERTAIN CONCEPTS. AND THEN THROUGHOUT THE PLAYBOOK, WE ALSO HAVE EQUITY CHECKS. SO THESE ARE MEANT TO PROMPT CONVERSATION WITH, UM, THE STAFF THAT ARE ENGAGED IN THE PROJECT, MEANT TO ASK QUESTIONS THAT INVOKE CONVERSATION ABOUT EQUITY. SO ARE WE DESIGNING FOR EQUITY? DID WE ENGAGE A REPRESENTATIVE SAMPLE OVER INTENDED USERS? IS OUR PROTOTYPE ACCESSIBLE TO DISABLED USERS? STUFF LIKE THAT. AGAIN, TO JUST HELP ENSURE THAT EQUITY IS REFLECTED THROUGHOUT THE PROCESS. AND THEN LASTLY TOO, WE HAVE THIS, UH, SHAREPOINT PAGE THAT I CREATED. SO, AS I WAS MENTIONING THOUGH, THOSE WERE JUST FOR EXAMPLES. THERE ARE MANY TOOLS, MANY OF THESE EXAMPLES CONTAINED THROUGHOUT THE PLAYBOOK. SO FOR EASIER ACCESS, UM, THERE IS A LINK IN THE, UH, PLAYBOOK, UM, TO GET TO THE SHAREPOINT PAGE. SO ALL OF THOSE EQUITY CHECKS, EXAMPLES AND ACTION TEMPLATES TOOLS ARE ALL ORGANIZED INTO THESE RESPECTIVE FOLDERS. AND PEOPLE CAN JUST CLICK, UH, WITHIN THE FOLDERS TO FIND, UM, WHATEVER TOOL THAT THEY NEED. AND THAT WAY THEY CAN MORE EASILY FIND IT. AND ALSO, IF THEY NEED TO COPY AND PASTE IT, IT'S A LOT EASIER TO DO IT THIS WAY, HAVE 'EM IN THEIR OWN WORD DOCS THAN HAVING TO SIFT THROUGH, UM, THE PLAYBOOK ITSELF, WHICH CAN BE PRETTY LONG. AND THEN ON THE RIGHT SIDE HERE, I ALSO HAVE JUST FOR EASIER ACCESS, ALL THE, UM, SOME RESOURCES, AGAIN, DIGITAL PLAYBOOK, UM, OTHER WEBSITES TO OTHER CITIES AND OTHER HELPFUL RESOURCES AND TOOLS FOR ADDITIONAL READING FOR STAFF. IN TERMS OF NEXT STEPS, UM, THE TECHNOLOGY INNOVATION DEPARTMENT IS WORKING TO FINALIZE THE CONTENT OF THE PLAYBOOK, RIGHT? FINISHING UP, INCORPORATING THE, UH, FEEDBACK THAT WAS RECEIVED FROM CITY STAFF, FROM TID LEADERSHIP TEAM AND THE FOCUS GROUPS, UM, AND WORKING ON CREATING, UM, THAT FINAL VERSION. UH, THE NEXT STEP, UM, IS TO GROUND TRUTH THE PLAYBOOK. UM, SO ONCE THE, UM, FINALIZED PLAY, THE CONTENT OF THE PLAYBOOK HAS BEEN FINALIZED IS TAKING THAT PLAYBOOK AND ACTUALLY USING IT ON A DIGITAL SERVICE PRODUCT TO CONDUCT CIVIC USER TESTING. UM, AND THAT WAY WE CAN ANALYZE THE EFFECTIVENESS OF THE PLAYBOOK AND MAKE ADJUSTMENTS AS NECESSARY. AND THEN ALONG WITH ROLLING OUT, UM, THE PLAYBOOK, UM, THERE'S ALSO GONNA BE A ROLLOUT OF A CUT AWARENESS TRAINING ACROSS CITY DEPARTMENTS. SO THIS TRAINING WILL ALSO HELP STAFF REALIZE AND UNDERSTAND THE VALUE, THE SIGNIFICANCE AND IMPORTANCE OF, UM, USING CIVIC USER TESTING TO ADVANCE EQUITY AND ACCESSIBILITY, BECAUSE THIS IS AN EQUITY ACCESSIBILITY ISSUE. AND THAT CONCLUDES MY PRESENTATIONS. UM, I'M HAPPY TO ANSWER ANY QUESTIONS THAT YOU MAY HAVE AT THIS TIME. UM, I DON'T HAVE ANY QUESTIONS, BUT I JUST WANNA SAY THANK YOU. I THOUGHT IT WAS REALLY THOROUGH AND, UM, SEEMS LIKE A GREAT RESOURCE FOR THE CITY. SO THANKS. THANK YOU. I'LL PIGGYBACK ON THAT, THAT WAS PRETTY DETAILED AND GRANULAR, AND I WANNA HIRE YOU . I'M JUST KIDDING. , THANK YOU SO MUCH. I APPRECIATE IT. UH, YEAH, THANK YOU VERY MUCH FOR THAT PRESENTATION. UM, I, I WAS HAPPY TO SEE HOW THE CONTENT EXTENDS BEYOND JUST, UM, NECESSARILY DIGITAL AND TO LIKE TESTING, LIKE THE EXAMPLE OF THE SPACE IN HARVEY MILK PLAZA, WHERE IT SEEMS LIKE IT WAS MORE PHYSICAL EQUIPMENT BEING TESTED IN THAT CASE. AND SO THAT WAS, UH, REALLY INTERESTING TO SEE IS, IS THAT, AM I UNDERSTANDING THAT CORRECTLY AS A QUESTION? YES. UM, THAT PARTICULAR PROJECT THAT WAS USED TO CIVIC USER TESTING WAS USED TO TEST OUT THE EQUIPMENT THAT THEY WOULD USE TO ACTIVATE THAT SPACE. VERY COOL. AWESOME. WELL, THANK YOU VERY MUCH. OF COURSE. THANK YOU. YEAH, AND I JUST WANTED TO ADD THAT, YOU KNOW, WE HAVE BEEN DOING FORMS OF CIVIC USER TESTING, FOR EXAMPLE, THAT HARVEY MILK PLA PLAZA, UM, EXAMPLE, OR OUR DIGITAL RIGHTS PLATFORM, OR MORE RECENTLY WE'VE BEEN DOING CIVIC USER TESTING WITH A, UM, SINGLE SIGN-ON PILOT PROJECT. UM, BUT WHAT I REALLY LIKE ABOUT THIS PLAYBOOK IS IT'S PUTTING IT ALL IN ONE PLACE, AND THEN WE'RE WITH THE, AFTER DOING THE GROUND TRUTHING AND, UM, EVERYTHING, WE CAN ROLL IT OUT TO ALL THE DEPARTMENTS. AND OUR HOPE IS THAT IT GOES WAY BEYOND DIGITAL SERVICES. AND IT'S, AND THE METHODOLOGIES AND THE PRINCIPLES AND, UH, THAT ARE BEING USED, UM, CAN BE REALLY HELPFUL TO OUR [00:30:01] CITY TEAM TO BE ABLE TO MAKE SURE THAT THEY'RE DESIGNING, UM, PRODUCTS AND SERVICES, UH, WITH COMMUNITY, NOT JUST FOR COMMUNITY. SO WE'RE REALLY EXCITED, AND I FEEL LIKE AMI DID A, A WONDERFUL JOB, UM, WITH THE ASSIGNMENT. AND ONE THING, UM, AS COMMISSIONERS, UH, WE ARE ALWAYS LOOKING FOR CIVIC USER TESTERS. SO WE CAN BE, WE CAN FOLLOW UP WITH, UM, INFORMATION, WE CAN SEND YOU THE LINK TO THE PLAYBOOK ITSELF IF YOU WANNA PROVIDE ANY MORE FEEDBACK. BUT ALSO, UM, WITH, UH, A LINK TO HOW WE'RE RECRUITING FOR CIVIC USER TESTING. AND IF YOU HAVE ANY, IF YOU, AS YOU RUN INTO RESIDENTS, UM, OR BUSINESS OWNERS, UM, YOU KNOW, IN THE COMMUNITY AND YOU THINK THAT THEY WOULD LIKE TO BE INVOLVED, THERE'S A, A SIGNUP PROCESS. AND THEN, UM, AS PROJECTS AND, AND SERVICES, UM, GET ROLLED OUT, WE WOULD, WE'LL LOOK AT THE, THE, THE CANDIDATE POOL OF CIVIC USER TESTERS AND IDENTIFY THEM. AND RYAN, DID YOU WANNA SAY ANYTHING ELSE ABOUT THAT? YEAH, THANKS LEAH. I WAS GONNA MENTION THAT. SO WE ACTUALLY HAVE ALREADY DEVELOPED A INTEREST FORM FOR CIVIC USER TESTERS THAT WAS RELEASED I THINK IN SEPTEMBER OR OCTOBER OF THIS YEAR. WE HAVE ABOUT 150 RESIDENTS WHO HAVE ALREADY SIGNED UP, AND WE'VE ALREADY ACTUALLY PULLED FROM THAT LIST FOR THE CIVIC USER TESTING OF THE SINGLE SIGN-ON PROTOTYPE, UH, THAT, THAT LEAH MENTIONED EARLIER. SO MOVING FORWARD, WE WOULD LOVE THE COMMISSION'S HELP WITH PROMOTING THAT SIGNUP FORM. UM, WE HAVE DIGITAL MATERIALS WE CAN SHARE. WE CAN ALSO SHARE PHYSICAL FLYERS THAT WE PUT TOGETHER AS WELL. IF YOU KNOW PEOPLE IN YOUR NEIGHBORHOODS OR, UH, OTHER, YOU KNOW, SOCIAL OR PROFESSIONAL CIRCLES THAT YOU THINK WOULD BE GOOD CIVIC USER TESTERS, UM, WE DO PLAN TO ROLL OUT CIVIC USER TESTING FOR FUTURE PROJECTS TOO, UH, ON AN ONGOING BASIS. I DO HAVE A QUESTION. SO EARLIER IT WAS MENTIONED THAT THERE WAS, UM, COMPENSATION FOR THE COMMUNITY MEMBERS. SO, UM, JUST CURIOUS ABOUT THAT PROCESS AND WHERE IS THAT BUDGET COMING FROM? RIGHT, SO FOR THE CIVIC USER TESTING FOR THE SINGLE SIGN-ON PILOT, WE ENDED UP DOING, CONDUCTING ABOUT 15 USER TESTS. UM, AND WE PROVIDED COMPENSATION IN THE FORM OF $25 GIFT CARDS FOR EACH TEST. UH, AND THAT WAS ADVERTISED ON THE FLYER AND THEN ALSO ADVERTISED TO, UH, PARTICIPANTS AND DIRECT EMAILS TO THEM. UM, THE BUDGET COMES FROM OUR OWN TECHNOLOGY AND INNOVATION DEPARTMENT. I THINK IT REFLECTS THE PRIORITIES OF THE DEPARTMENT THAT WE CARE ABOUT THIS, AND WE THINK IT'S AN IMPORTANT PART OF, UH, DEVELOPING EQUITABLE AND ACCESSIBLE PRODUCTS. COMMISSIONER WOODMAN NANCE, UM, YES, I WANTED TO SAY THE, THE PRESENTATION WAS, UM, OUTSTANDING. I, I DO HAVE A QUESTION. I KNOW IT HAS BEEN LAID OUT WHAT THE NEXT STEPS ARE. I WAS JUST CURIOUS, IS THERE ANY TYPE OF TIMELINE IDENTIFIED AS TO WHEN THESE ARE GONNA BE CARRIED OUT? YEAH, I, I CAN TRY TO TAKE THAT ONE. SO WE'RE ACTUALLY IN THE FINAL PROCESSES OF FINALIZING THE PLAYBOOK CURRENTLY. SO I EXPECT WE'LL HAVE IT PUBLISHED ON OUR WEBSITE AND AVAILABLE TO CITY STAFF WITHIN THE NEXT MONTH. UH, AT THAT POINT, WE CAN SHARE THE LINK WITH YOU SO YOU CAN ACCESS IT, UH, AS WELL AS THE DIGITAL MATERIALS. UM, WE'RE ALSO IN THE PROCESS OF COORDINATING THE C-U-C-U-T AWARENESS TRAINING THAT AMI MENTIONED AS WELL, UH, POTENTIALLY WORKING WITH AN EXTERNAL PARTNER FOR THAT. UM, OUR VISION FOR THAT IS RATHER THAN JUST HAVING LIKE A WEBINAR WHERE WE INVITE ALL STAFF, UM, WE WANNA ACTUALLY WALK THROUGH THE CUT PROCESS WITH STAFF. SO WE'RE ACTUALLY TRYING TO FIND INTERNAL STAFF WHO HAVE A PROJECT THAT THEY'RE KICKING OFF AND HAVE THEM GO THROUGH THE DIFFERENT PHASES OF THE CUT PROCESS TO LEARN THE PLAYBOOK AND TO LEARN HOW IT WORKS. UM, I DON'T HAVE A TIMELINE FOR WHEN WE WILL ROLL THAT OUT, BUT I EXPECT IN THE EARLIER PART OF 2024. UM, AND THEN, LIKE I MENTIONED EARLIER, WE ARE LOOKING TO INSTITUTE CUT ON AN ONGOING BASIS FOR UPCOMING, YOU KNOW, PROJECTS AND SERVICES. UH, I KNOW ONE THING THAT HAS BEEN DISCUSSED IS, IS POSSIBLY ROLLING IT OUT FOR THE GO LONG BEACH APP. UM, I THINK WE NEED TO CONFIRM THAT WE CAN DO THAT, BUT THAT WOULD BE A GREAT CANDIDATE, I THINK, TO IMPROVE THE ACCESSIBILITY AND USABILITY OF IT. THANK YOU. DO WE HAVE PUBLIC COMMENT FOR ITEM NUMBER THREE C? NONE. PLEASE CAST YOUR VOTES. MOTION CARRIES. ITEM NUMBER FOUR, RECOMMENDATION TO RECEIVE AND FILE A REPORT FROM CURRENT AD HOC COMMITTEES AND DISCUSS ANTICIPATED AD HOC COMMITTEES FOR UPCOMING PROJECTS. [00:35:03] ALL RIGHT, COMMISSIONERS, UM, FOR THIS ITEM, UM, WE FOR SURE WANNA HAVE A REPORT OUT FROM OUR AD HOC COMMITTEE ON, ON, UH, GENERATIVE AI, BUT THEN ALSO AS PART OF THIS ITEM, WE CAN TALK ABOUT, [4. 23-51627 Recommendation to receive and file a report from current Ad Hoc Committees and discuss anticipated Ad Hoc Committees for upcoming projects.        Suggested Action: Approve recommendation. ] UM, SOME OTHER POTENTIAL AD HOC COMMITTEES IN WORK, YOU KNOW, AHEAD FOR THE COMMISSION. SO, BUT I THINK MAYBE FIRST WE CAN DO THE, THE GENERATIVE AI, UM, C COMMITTEE REPORT OUT, AND I BELIEVE, UH, JOHN OR COMMISSIONER OR DE ARMES IS PLANNING TO DO THAT. YES. UM, SO, SO, UH, THANK YOU. UH, WE, UH, STARTED THE AD HOC COMMITTEE FOR GENERATIVE AI, UH, GUIDELINES, UM, BACK IN AUGUST. AND, UH, WE HAVE, UM, AS A GROUP, WE HAD COMMISSIONER, UH, NICOLE CALLAHAN, UH, JAVIER ORTIZ AND MYSELF WORKING WITH, UM, OMAR RYAN AND, UH, JONATHAN FROM, UH, UM, TID. AND, UH, SO THANKS, THANKS TO, UH, LEAH AND THE REST OF THE TEAM FOR HELPING GATHERING A BUNCH OF, UH, EXAMPLES FROM OTHER MUNICIPALITIES ACROSS THE COUNTRY. UH, WE LOOKED AT STUFF FROM SEATTLE AND BOSTON AND, UM, LET ME SEE MY NOTES HERE. UM, WE LOOKED AT SOME, EVERYTHING FROM ACTUAL, UH, POLICIES OR GUIDELINES OR CHEAT SHEETS THAT WERE DISTRIBUTED IN VARIOUS PLACES OVER THE LAST, UH, YEAR, BASICALLY SINCE THE ADVENT OF THE CHAT GPT REVOLUTION. UM, AND WHAT WE'VE DONE IS COMPILED A DOCUMENT, THANK YOU VERY MUCH TO OMAR, UH, FOR DOING A LOT OF THE HEAVY LIFTING THERE. UM, BUT, UH, WE HAVE A DOCUMENT HERE THAT IS BASICALLY DRIVEN BY THE PRINCIPLES OF THE, UH, CITY OF LONG BEACH, THE SAME PRINCIPLES THAT ARE, UH, SENT AROUND IN CITY, UM, DOCUMENTATION EVERY DAY. UH, WE HAVE A LITTLE BIT OF A HIGH LEVEL GUIDELINES. WE'RE NOT TRYING TO SET POLICY, BUT REALLY JUST ACKNOWLEDGE THAT WE'RE IN A TIME WHERE GENERATIVE AI IS BRAND NEW, AND IT'S USED IN VARIOUS WAYS BY VARIOUS FOLKS, AND IT'S ON THE PATH TO BECOMING SOMETHING LIKE A SPELLCHECKER IN THE FUTURE, BUT WE'RE NOT QUITE THERE YET. SO THIS DOCUMENT IS DESIGNED TO GIVE, UH, GUIDELINES ON BEST PRACTICE, SAFE USE, UH, AND THEN, UH, THERE ARE EIGHT POINTS IN THE GUIDELINES. AND THEN WE GIVE SOME SAMPLE USE CASES, ACTUAL PHYSICAL EXAMPLES OF, HERE'S STUFF YOU MIGHT PUT INTO SOMETHING LIKE CHAT GPT. AND SO IT'S SOMETHING YOU MIGHT GET OUT. UM, I'M NOT SURE, UH, WHERE WE ARE WITH SHARING THE DOCUMENT WITH THE REST OF THE COMMITTEE. I THINK I'LL ASK RYAN IF HE CAN HELP ME ON THAT. I, I DID, UH, A COUPLE TIMES, UH, FAIL TO GIVE MY OWN LITTLE UPDATE ON A VISUAL DESIGN, UH, DRAFT OF IT. BUT, UH, THE CONTENT IS, UM, VERY MUCH AGREED UPON, UH, BY THE AD HOC COMMITTEE AT THIS TIME. UM, RYAN, DID YOU HAVE ANYTHING TO ADD? SURE, YEAH, THANK YOU COMMISSIONER FOR THAT REPORT OUT. UM, IT'S BEEN WONDERFUL WORKING WITH YOU AND THE OTHER TWO COMMISSIONERS ON THE DEVELOPMENT OF THE, UM, GENERATIVE AI INTERIM GUIDANCE. JUST A COUPLE OTHER THINGS I WANTED TO TOUCH ON. SO IN ADDITION TO REVIEWING BEST PRACTICES, UH, THE CITY ALSO ISSUED A STAFF SURVEY, UH, THAT WE DISTRIBUTED CITYWIDE TO UNDERSTAND KIND OF A BASELINE OF HOW STAFF ARE USING OR ARE NOT USING GENERATIVE AI. UM, WE GOT ABOUT 700 SURVEY RESPONSES, UM, WHICH WAS REALLY GREAT. AND, UM, THAT KIND OF HELPED US INFORM HOW WE WOULD DIRECT THE POLICY AND THE GUIDANCE TO ADDRESS SOME OF THE USE CASES THAT STAFF WERE USING IT FOR ALREADY, OR WE'RE EXCITED ABOUT USING. UM, AND THE AI GUIDANCE HAS, I THINK, EIGHT COMPONENTS. UM, WE HAVE OUR GUIDELINES, WHICH INCLUDE THINGS LIKE, UM, YOU KNOW, DO NOT PUT IN ANY PERSONAL OR SENSITIVE INFORMATION, UM, CAREFULLY REVIEW AND FACT CHECK, UH, OUTPUT FROM GENERATIVE AI, THINGS LIKE THAT. WE HAVE SOME SAMPLE USE CASES ABOUT HOW STAFF MIGHT ACTUALLY USE IT FOR THINGS LIKE DRAFTING, UH, AN INTERNAL COMMUNICATIONS EMAIL, ANALYZING DATA SETS ALONG WITH THE RISK LEVEL. FOR EACH OF THOSE USE CASES. WE HAVE PRINCIPLES WHICH COMMISSIONER DE ARMAS ALLUDED TO. UH, AND THEN A COUPLE PRACTICAL SECTIONS LIKE HOW TO CITE YOUR USE OF GENERATIVE AI, HOW TO OPT OUT OF YOUR DATA BEING TRACKED IN GENERATIVE AI, UH, A RISK MATRIX OF THE USE CASES. AND THEN JUST CLOSING WITH SOME TERMS, UM, AND DEFINITIONS. AND THE LAST THING I'LL SAY IS WE TRIED TO STRUCTURE THIS DOCUMENT TO BE LESS OF AN ADMINISTRATIVE REGULATION, SOMETHING VERY OFFICIAL, BUT MORE USABLE, MORE USER-FRIENDLY AND MORE FOCUSED ON EMPOWERING THE RESPONSIBLE USE OF GENERATIVE AI WHILE THE CITY [00:40:01] CONTINUES TO EXPLORE SOME OF THE RISKS AND BENEFITS. ABSOLUTELY. THANK YOU. SO ARE THERE ANY, UM, COMMENTS FROM THE COMMISSIONERS? SO THE IDEA IS THAT YOU'LL SEND A DRAFT TO THE COMMISSION AT SOME POINT, PRE-DESIGN. I MEAN, I'M COMFORTABLE IF YOU WANNA WAIT TILL IT'S DESIGNED, BUT I'M ALSO OKAY SEEING THE, I DON'T THINK THERE'S ANY REASON FOR ME TO HOLD IT BACK, BUT, UM, WE'RE THE, THE VERSION WE HAVE IN, UM, IN SHAREPOINT RIGHT NOW IS, UH, TAGGED FINAL. SO WE WILL, UM, WHATEVER THE STANDARD WAY TO DISTRIBUTE STUFF LIKE THAT, UM, WITHOUT TRIGGERING UNNECESSARY MEETINGS. WE'LL, WE'LL DO THAT. YEAH, WE CAN, WE CAN DEFINITELY SEND AN EMAIL OUT TO ALL THE COMMISSION. UM, AND WHAT WE COULD DO IS THEN IF AND COMMISSIONS HAVE ANY INDIVIDUAL FEEDBACK THEY WOULD LIKE TO GIVE, THEY CAN JUST RESPOND DIRECTLY TO, UM, OUR EXECUTIVE SECRETARY ALEJANDRA. UM, AND, AND THEN THE NEXT STEPS IS ONCE WE, AND AS RYAN AND, UM, COMMISSIONER DEIS MENTIONED, YOU KNOW, THIS IS INTENDED TO BE, UH, EVOLVING, YOU KNOW, AND SO WE DO WANNA GET IT OUT THERE BECAUSE THIS IS, UH, UH, SOMETHING THAT, YOU KNOW, THE, THE STAFF IS VERY EXCITED ABOUT, BUT THERE'S ALSO OUR, WE, WE LEARNED OUR EMPLOYEES ARE ALSO VERY CONCERNED ABOUT IT. SO WE DO WANNA GET THESE GUIDELINES OUT THERE SOONER RATHER THAN LATER. AND SO OUR HOPE IS TO HAVE 'EM GO TO OUR, YOU KNOW, GET ISSUED TOMORROW OR ISSUED TO THE, TO CITY LIKE NEXT MONTH IN JANUARY. SO WHEN WE'LL SEND IT OUT, GET SOME HOPE, HOPEFULLY YOU WILL GET SOME FEEDBACK IN THE NEXT WEEK OR TWO. AND THEN WE CAN FINALIZE THE, YOU KNOW, THE FINAL INTERIM GUIDANCE AND IT'S NEVER FINAL, BUT, UM, AND THEN SEND IT OUT TO, TO THE CITY STAFF. AND, UM, AND THEN THE OTHER THING WE'LL BE PLANNING ON DOING IS HAVING FOCUS GROUPS WITH, WITH THE CITY STAFF TO LIKE FURTHER EXPLORE THE GENERATIVE AI USE CASES. AND THE OTHER THING I WANTED TO ALSO MENTION IS, IN ADDITION TO THIS WORK THAT WE'RE DOING HERE IN LONG BEACH, WE'RE PART OF A COUPLE OF COALITIONS, UM, OF OTHER CITIES AND ENTITIES EXPLORING GENERATIVE AI. SO WE'RE, WE'RE NOT ALONE IN THIS WORK, AND WE'RE VERY MUCH LEARNING FROM WHAT OUR OTHER, WHAT OTHER CITIES ARE DOING, BUT WE'RE ALSO MAKING SURE THAT IT WORKS FOR LONG BEACH. AND I'M ESPECIALLY, I'M ESPECIALLY PROUD OF THE PRINCIPLES, UH, THAT ARE IN PART OF THIS DOCUMENT BECAUSE THAT INCLUDES THINGS LIKE EMPOWERMENT, EXPLORATORY MINDSET, TRANSPARENCY AND ACCOUNTABILITY, EQUITY AND INCLUSION, AND, UH, SAFETY AND SECURITY. SO ALL OF THAT COLLECTIVELY WORKS TOGETHER TO, TO SIGNIFY LIKE THE INTENT FOR THE, THE, THE BENEFITS OF GENERATIVE AI BY WHILE ALSO EVALUATING THE RISKS OF GENERATIVE AI AND BEING TRANSPARENT IN OUR USE. YEAH. UM, THANK YOU FOR THAT. I HAVE A QUESTION. UM, WITH REGARD TO, UM, ASSESSING RISKS, UM, AND DESIGNING PRINCIPLES, ARE YOU ALSO THINKING ABOUT, UM, ETHICAL USE CASES, UM, AND, AND THE PLATFORMS THAT ARE COLLECTING DATA AND TRAINING USING DATA THAT THEY'RE TRAINING THESE MODELS OFF OF, THAT THEY'RE ACTUALLY ETHICAL AND THAT, AND IF THERE'S AN OPPORTUNITY TO REEVALUATE AT SOME POINT, LIKE, HEY, WE JUST LEARNED CHAT. GBT IS BEING SUED BY THE NEW YORK TIMES BECAUSE THEY'RE USING DATA, UM, ILLEGALLY, RIGHT, TO TRAIN THEIR MODELS. AT WHAT POINT DO WE DRAW A LINE WITH REGARD TO UTILIZING CERTAIN AI TECHNOLOGIES, UM, WHEN THEY ARE CAUSING HARM OR DOING THINGS UN UNETHICALLY OR ILLEGALLY? YEAH, THAT'S A GREAT QUESTION. AND UM, LEAH MENTIONED THIS GOVERNANCE OR YEAH, GOVERNANCE OF AI COALITION THAT THE CITY IS A PART OF, ALONG WITH I THINK 50 OTHER MUNICIPALITIES THROUGHOUT THE COUNTRY. UM, ONE OF THE, A COUPLE OF THE DELIVERABLES THAT THEY'RE WORKING ON ADDRESSED THE QUESTIONS THAT YOU RAISED. SO THEY'RE WORKING ON DEVELOPING AN AI VENDOR REGISTRY, BASICALLY A SHARED DATABASE OF INFORMATION ABOUT AI VENDORS. UM, THEY'RE WORKING ON DEVELOPING AN AI FACT SHEET THAT WOULD BE INCLUDED IN A CITY'S OR AN AGENCY'S, UM, PROCUREMENT PROCESS THAT WOULD BE REQUIRED FOR THE VENDOR TO FILL OUT. THEY'RE ALSO WORKING ON SOME DRAFT CONTRACT LANGUAGE THAT WOULD BE INCLUDED IN, IN CONTRACTS WITH VENDORS THAT WOULD STIPULATE THINGS JUST LIKE YOU MENTIONED, LIKE, UH, YOU KNOW, ACCOUNTABILITY STANDARDS FOR VENDORS USE OF AI, UH, HAVING THEM SHARE REPORTS ABOUT HOW THEIR TRAINING MODELS WERE VETTED, UH, AND THINGS LIKE THAT. SO WE'RE CONTRIBUTING TO THOSE DISCUSSIONS [00:45:01] AT THE NATIONAL LEVEL, AND I THINK WE PLAN TO USE A LOT OF THOSE DELIVERABLES HERE AS WELL AND INSTITUTE THOSE INTO OUR, UM, OUR VENDOR CONTRACTING PROCESSES. YEAH. UH, TO ADDRESS THAT ALSO, UM, WHEN YOU GET THE DRAFT IN THE PRINCIPLE SECTION, THE SECOND BULLET POINT UNDER EQUITY AND INCLUSION, UM, MAYBE WE CAN, UH, RAISE THIS LANGUAGE A LITTLE BIT, BUT, UH, EVERYTHING WE DO REGARDLESS OF THE TOOLS IS THE REFLECTION OF THE CITY AND OURSELVES. WE ACKNOWLEDGE THAT AI SYSTEMS HAVE THE POTENTIAL TO PERPETUATE INEQUITIES, DISCRIMINATION, HARM AND BIAS TO LONG BEACH RESIDENTS. AND THE CITY WILL COMMIT TO USING RACIAL EQUITY LENS TO EXAMINE THE BURDENS, BENEFITS AND UNINTENDED CONSEQUENCES OF GENERATIVE AI. THAT'S THE CLOSEST PART THAT I THINK TOUCHES WHERE YOU'RE AT, BUT I I THINK IT MIGHT BE TWEAKED EVER SO SLIGHTLY TO REALLY NAIL IT, HUH? YEAH. YEAH. COOL. I LOOK FOR, I LOOK FOR, DO WE HAVE PUBLIC COMMENT ON ITEM NUMBER FOUR? ACTUALLY, WE STILL, I, WE'RE STILL, OH, SORRY ABOUT THAT. BUT IF, IF WE'RE DONE WITH THE GENERATIVE AI, UM, COMPONENT. OKAY. SO YES. THAT, THAT'S AN EXAMPLE OF AN AD HOC COMMITTEE THAT'S GOING WELL AND IT'S BEEN WORKING AS INTENDED AND MEETING WHAT, EVERY OTHER WEEK, I THINK WE'VE BEEN MEETING BIWEEKLY YEAH. FOR ABOUT THREE OR FOUR MONTHS NOW. YEAH. YEAH. AND SO, SO WE REALLY DO, UH, LIKE THE POWER OF THE AD AD HOC COMMITTEES BECAUSE THEY CAN BE, UM, AND JUST AS A FEW REMINDERS THEY ARE, THEY HAVE TO BE NO MORE THAN THREE MEMBERS. UM, AND NO FEWER THAN TWO . UM, 'CAUSE ONE PERSON'S NOT A COMMITTEE. UM, BUT YEAH. AND THEN THE WAY THEY'VE BEEN WORKING IS THEY, UM, THE STAFF, IT'S KIND OF STAFF WORKING WITH COMMISSIONERS AND GETTING, YOU KNOW, AND COMMISSIONERS MIGHT HAVE ASSIGNED, UM, DELIVERABLES OR PROJECTS OR THOUGHT THINGS TO DO BETWEEN MEETINGS, BUT THEN SOMETIMES IT, IT'S JUST AS IMP JUST AS IMPORTANT AS SHOWING UP AT THE MIDI THE MEETING AND PROVIDING INPUT AND, AND, AND INFORMING, UH, THE FINAL PRODUCTS. SO I JUST WANTED TO REMIND THE COMMISSION THAT IN ADDITION TO THIS, THIS AD HOC COMMITTEE, THE OTHER ONE, ONE THAT WE'RE LIKE WANTING TO LAUNCH, AND WE WOULD DO THAT FORMALLY IN THE NEXT ITEM IS THE DATA PRIVACY AD HOC COMMITTEE. AND LET ME, UM, BRING, PULL UP. YES. YEAH. AND I'LL TURN IT OVER TO RYAN TO JUST EXPLAIN, EXPLAIN THE PURPOSE OF THAT COMMITTEE. DO WE WANNA ADDRESS THAT ON ITEM FIVE, PERHAPS? OH, WELL, WELL, OKAY. SO WELL JUST FOR FOUR, WE NEED TO DISCUSS ANTICIPATED AD HOC COMMITTEES FOR UPCOMING PROJECTS. SO THAT'S THE ONLY ONE, I MEAN, OR, SO I GUESS THE OTHER, WELL, WE'LL TALK ABOUT THE DATA PRIVACY MORE ON ITEM FIVE, BUT IN TERMS OF THE OTHER, I'M TRYING TO FIND, OH YEAH, IF YOU'RE ASKING FOR AN UPDATE ON SOME OF THE OTHER AD HOC COMMITTEES, WE ANTICIPATED IT WAS THE DATA PRIVACY AD HOC COMMITTEE. AN AD HOC COMMITTEE FOCUSED ON DEVELOPING A COMMUNITY TECH FESTIVAL AND THEN AN AD HOC COMMITTEE FOR, TO SUPPORT TECHNOLOGY NEEDS FOR URGENT CITY PRIORITIES, SUCH AS THE CLIMATE ACTION PLAN, THE HOMELESSNESS CRISIS IN THE 2028 OLYMPICS. YEAH. AND THEN IN ADDITION TO THE AD HOC COMMITTEES, THERE'S, JUST AS A REMINDER, THERE'S ALSO AN ASSIGNMENT RELATED TO DIGITAL INCLUSION. AND WE HAVE TWO MEMBERS OF THE COMMISSION WHO ARE ASSIGNED TO THE DIGITAL INCLUSION STAKEHOLDER IMPLEMENTATION COMMITTEE. AND THAT COMMITTEE IS PROBABLY GONNA BE, UM, RE UH, THAT COMMISSION, THE DIGITAL INCLUSION STAKEHOLDER IMPLEMENTATION COMMITTEE IS GOING TO BE, UH, REFORMING NEXT MONTH OR FEBRUARY. SO THE NEXT MEETING IS GONNA BE HELD PRETTY SOON. AND THEN LASTLY, UM, THE OTHER, WE'RE NOT, THIS ISN'T READY FOR A COMMISSION A COMMITTEE YET, BUT THE, WE ARE DOING TECHNOLOGY STRATEGIC PLANNING, AND THAT'S ALSO GETTING KICKED OFF NEXT MONTH. AND SO WE'RE NOT READY YET FOR THE COMMISSION TO BE INVOLVED IN THAT PROCESS, BUT THERE'LL BE, THERE'LL BE STEPS WHERE THE COMMISSION GETS INVOLVED IN THAT PROCESS. SO I, I JUST WANTED TO LET EVERYONE KNOW WHAT KIND OF THE INTENDED WORK IS. SO AS WE TALK ABOUT FORMING THE AD HOC COMMITTEES AND THEN WHO'S, UH, MEMBERS OF THE AD HOC COMMITTEES, YOU KIND OF KNOW WHAT'S OUT THERE. SO IS THERE ANY KIND OF QUESTIONS ABOUT POTENTIAL AD HOC COMMITTEES FOR ITEM FOUR? YEAH, I DO. I, I HAVE SOME QUESTIONS AND I WANNA ALSO, I GUESS, SET EXPECTATIONS SO THAT WE AS A COMMISSION CAN KNOW WHAT WE'RE SIGNING UP FOR AND THAT WE ARE ACCOUNTABLE TO EACH OTHER. BECAUSE I KNOW SEVEN YEARS IN THIS, FOR MY PERSONAL EXPERIENCE, [00:50:01] SOMETIMES PEOPLE, SOMEBODY'S DOING MORE WORK THAN THE OTHER, OR PEOPLE DON'T SHOW UP. AND IS THERE AN OPPORTUNITY WHERE THEY, SOMEBODY CAN BE REMOVED FROM AN AD HOC COMMITTEE SO THAT SOMEBODY ELSE WHO REALLY ALSO WANTS TO BE PART OF THAT CAN PARTICIPATE AND ACTUALLY CONTRIBUTE MEANINGFULLY. SO I WANNA ESTABLISH THOSE KIND OF GUIDELINES OR UNDERSTANDING OF LIKE, WHAT ARE THE ROLES AND EXPECTATIONS ONCE YOU'RE IN THE AD HOC, UH, UH, COMMITTEE AND WHAT ARE, WHAT DOES THE, UH, TRANSITION LOOK LIKE IF, IF ONE NEEDS TO BE MADE. THE OTHER THING IS, I FEEL LIKE, BECAUSE WE DID A LOT OF THIS REALLY DEEP WORK IN THE BEGINNING OF THE YEAR WHERE WE TALKED AND OUTLINED, WE TALKED ABOUT AND OUTLINED WHAT OUR INTERESTS WERE, UM, AND WHITEBOARDED IT OUT AND, AND EVERYTHING. AND I FEEL LIKE, I THINK, I DON'T KNOW ABOUT YOU GUYS, BUT I FEEL LIKE WE NEED TO KIND OF REVISIT SOME OF THAT SO THAT WE CAN ACTUALLY IMPLEMENT, UM, THAT WORK AND THE, OUR CONTRIBUTIONS INTO THE NEW YEAR. UM, I DON'T KNOW WHAT THAT LOOKS LIKE ON THE AGENDA OR HOW TO ADD IT, BUT I FEEL LIKE THAT WOULD BE MEANINGFUL FOR US TO BE ABLE TO SAY, OKAY, HERE'S OUR, HERE'S THE COMMITTEES WE'RE GONNA DO, HERE'S THE WORK WE'RE GONNA DO. 'CAUSE I'VE SIGNED UP FOR LIKE THREE AD HOC COMMITTEES, BUT I'VE BEEN ON NONE. I MEAN, I'VE DONE NO WORK. SO I KIND OF WANT TO FIGURE OUT HOW WE CAN GET THERE AS A STARTING POINT. OH YEAH, CHAIR BRUNSON, THOSE ARE GREAT QUESTIONS AND OBSERVATIONS. UM, WHAT I THINK WE CAN DEFINITELY DO FOR NEXT MONTH IS, AND THAT'S A GREAT, JANUARY IS A GREAT NEW YEAR, UM, IS, IS AGENDAS ARE THE 2024 WORK PLAN AND BRING BACK WHAT WE HAD TALKED ABOUT WHEN WE WERE DOING THIS WORK IN 2023. AND ONE OF THE, ONE OF THE, UM, CONSTRAINTS ABOUT, UH, THE ACTUAL FORMATION OF THESE AD HOC COMMITTEES IS THEY GET FORMED BY THE COMMISSION AS A WHOLE. AND WE'D ONLY SET UP THE, UH, ONE OF OUR LESSONS LEARNED IS WE HAD ONLY SET UP THE GENERATIVE AI AD HOC COMMITTEE IN OUR JULY MEETING, AND THEN WE WEREN'T, BECAUSE WE HAVEN'T MET, WE HAVEN'T BEEN ABLE TO FORMALLY LAUNCH THE, THE OTHER ONES. SO THAT'S WHAT WE WANNA DO TODAY. AND THAT'S THAT WHY WE DEFINITELY WANNA MAKE SURE WE WE'RE LAUNCHING THE DATA PRIVACY AD HOC COMMITTEE. UM, AND IF WE WANNA START LAUNCHING SOME OF THE OTHER ONES TOO, EVEN IF WE'RE NOT QUITE READY YET, THEN RELEASE, THEY'RE OFFICIALLY LAUNCHED. AND THEN WE CAN, UM, YOU KNOW, START CONVENING IN TERMS OF ATTEN, LIKE ATTENDANCE REQUIREMENTS AND OTHER THINGS. WE UNDERSTAND THAT ALL OF YOU ARE VOLUNTEERS, BUT IT, IT DOES, BECAUSE THESE COMMITTEES HAVE TO BE NO MORE THAN, UM, THREE MEMBERS. IT WOULD BE GOOD TO HAVE, UM, TO TALK ABOUT, IT MAY BE IT'S SELF-GOVERNANCE WHERE, YOU KNOW, A VOLUNTEER SAYS, YOU KNOW WHAT? I HAVEN'T COME TO ANY OF THESE MEETINGS. I, I PROBABLY NEED TO STEP DOWN FROM THE AD HOC COMMITTEE, AND THEN WE CAN CHANGE THE MEMBERS THROUGH THESE, THE ACTION AT THESE ITEMS, YOU KNOW, ON THE AGENDA. AND BY THE WAY, ITEM FOUR AND FIVE ARE GONNA BE ON EVERY AGENDA. IT'S INTENDED TO HAVE A, YOU KNOW, HEY, REPORT OUTS OF AD HOC COMMITTEES, YOU KNOW, HAVE A DISCUSSION ABOUT HOW THEY'RE GOING AND THEN FORMALLY CREATE OR ABOLISH THE AD HOC COMMITTEES BASED ON THE DISCUSSION ON ITEM FOUR AND ITEM FIVE. THAT'S HOW IT'S INTENDED TO, TO BE. THANK YOU. SORRY LISA. LEMME TAKE STEP ON YOUR TOES. UM, SO JUST IN THE INTEREST OF THEN GETTING IT STARTED, TO YOUR POINT, WOULD IT BE HELPFUL IF WE MOVE TO START, WE'RE GONNA TALK ABOUT DATA PRIVACY IN A SECOND. SO STRATEGIC PLANNING AND THE CITY PRIORITIES, I FORGET WHAT THAT COMMITTEE ONE WAS CALLED. RYAN, YOU SAID, SO IF WE SAY IT TODAY AND MAKE THAT MOTION, THEN WE CAN KICK OFF THAT WORK. IS THAT RIGHT? OR AT LEAST BE ESTABLISHED SUCH THAT WE COULD WORK ON IT IN SUBSEQUENT COMMITTEE MEETINGS? YEAH, AND WE CAN ALSO, WE CAN DO ANY, UM, WE CAN REALLY, YOU KNOW, AND ACTUALLY OUR CITY ATTORNEY'S HERE, BUT WE CAN, WE CAN REALLY TALK ABOUT WHAT'S, WHAT'S POSSIBLE AND FOUR, AND THEN DO THE FORMAL MOTIONS IN FIVE. THAT'S WHERE, THAT'S KIND OF WHERE I THINK WE'RE AT. YEAH. SO AT THIS POINT, UM, FOR SURE WE KNOW WE WANNA, WE'LL LAUNCH THE DATE PRIVACY, UM, AD HOC COMMITTEE. WAS THERE ANY OTHER ONES WE WANTED TO START? I THINK ANOTHER ONE THAT COULD BE WORTH STARTING IS THE, I FORGET WHAT WE CALLED IT HERE, BUT THE TECHNOLOGY STRATEGIC PLANNING, UM, AD HOC COMMITTEE TO SUPPORT US WITH OUR TID 25, WHICH IS A SHORT TERM IT STRATEGIC PLAN FOR OUR DEPARTMENT. UM, WE, WE DO HAVE A CONSULTANT THAT WE'RE WORKING WITH ON THAT PROJECT, AND WE HAVE A CITYWIDE KICKOFF PLANNED FOR JANUARY 24TH. [00:55:02] UM, I DON'T THINK WE ANTICIPATE THAT WE'LL BE STARTING OUR COMMUNITY OUTREACH IN OUR COMMUNITY ENGAGEMENT ON THAT UNTIL A LITTLE BIT LATER, BUT I THINK IT CAN'T HURT TO FORM THE SUB AD HOC COMMITTEE NOW, SO THE GROUP IS READY TO SUPPORT AS NEEDED. SO I THINK THAT'S A GOOD PLAN. OKAY. AND THEN NEXT MONTH WE'LL BRING BACK, UM, ALL THE WORK WE DID PREVIOUSLY AND THEN REVISIT IT BECAUSE WE DO HAVE NEW, UH, NEW COMMISSIONER AND ALSO PROBABLY WE MIGHT EVEN HAVE A NEW, NEW COMMISSIONER BECAUSE WE ARE, THE MAYOR IS IN COUNCIL, WOULD BE WORKING TO FILL THE SEVENTH SPOT AS WELL. SO, UM, DO WE HAVE TO, UH, AGENDIZE THAT OR M MAKE A MOVE TO AGENDIZE THAT? HOW DOES THAT WORK? UM, NO, WE, WE CAN PUT IT, I CAN DISCIPLINE THIS DISCUSSION. I CAN DEFINITELY HAVE IT PUT ON THE AGENDA. UH, THE, THE ONE QUESTION IS, DO WE WANNA MAKE IT IN A MORE, MORE INFORMAL STUDY SESSION OR THIS, BECAUSE, YOU KNOW, AS YOU REMEMBER, THE MORE INFORMAL STUDY SESSION IS HELD IN A DIFFERENT ROOM AND A LITTLE BIT MORE INFORMAL, BUT IT REALLY, WE COULD GO EITHER WAY. SO WOULD THAT BE SOMETHING THAT WE, WE WOULD BE ABLE TO SCHEDULE FOR EARLIER IN THE YEAR, EARLIER IN JANUARY BEFORE THE 24TH MEETING? IF WE HELD A STUDY, SPECIAL STUDY SESSION? WE COULD POSSIBLY DO THAT. IT WOULD, SO WE COULD HAVE OUR, OUR EXECUTIVE SECRETARY ALEJANDRA WORK ON TRYING TO GET PEOPLE TOGETHER BEFORE. ONE THING IS, I AM GONNA BE OUT ON THE FIRST TWO WEEKS OF JANUARY, SO IT, IT WOULD HAVE TO BE THAT THIRD WEEK. OKAY. I WAS TRYING TO, I HAVE KNEE SURGERY, SO I WON'T BE ON THE 20 ON THE, UH, AT THE, IN JANUARY. SO, UM, YEAH, I MEAN, I'M OPEN TO A STUDY SESSION. I FEEL LIKE IT'S, IF WE CAN CENTER THE WORK THAT WE'RE GONNA DO FOR THE YEAR, THAT WOULD BE, I'D LIKE TO SEE NEXT YEAR BE A MORE PRODUCTIVE YEAR FOR US. CAN WE, CAN WE WORK TO THEN, UM, GET A SCHEDULE GOING FOR, FOR A SPECIAL STUDY SESSION AHEAD OF THE REGULAR SESSION? YES. WE CAN WORK ON THAT. THANK YOU. AND JUST A QUESTION, SO IS A STUDY SESSION FOR THE STRATEGIC PLANNING AD HOC COMMITTEE, OR IS IT MORE FOR THE FULL YEAR PLANNING? UH, THE STUDY SESSION WOULD BE, SO YOU ATTENDED? YES. SO REMEMBER WHEN WE GOT TO LIKE REALLY TALK ABOUT OUR SKILL SETS, OUR INTERESTS, EVERYTHING THAT WE WANTED TO WORK ON AS A COMMISSION. MM-HMM. . UM, BUT WE NEVER REALLY GOT TO PUT THAT WORK INTO, INTO PLAY. MM-HMM. . SO I WANT US TO BE ABLE TO KIND OF REVISIT THAT, SEE WHAT STICKS, WHAT HOLDS TRUE MM-HMM, , UM, AND THEN REALLY CARVE OUT LIKE, OKAY, THIS IS, THIS IS THIS GROUP, THIS PROJECT, THIS GROUP, THIS PROJECT, THIS IS WHAT WE'RE GONNA WORK ON FOR THE YEAR. 'CAUSE WE, I, WE DIDN'T GET A CHANCE TO REALLY MAKE THAT STICK LAST YEAR. AND I'D LOVE THAT TO BE WHAT WE DO IN THIS NEXT YEAR. OKAY. ARE YOU GUYS GOOD WITH THAT? YEAH. OKAY. I THINK WE'RE DONE WITH THIS ITEM. THANK YOU. DO WE HAVE PUBLIC COMMENT ON ITEM NUMBER FOUR? SEE NONE MEMBERS PLEASE CAST YOUR VOTE. COMMISSIONER KELLY. THANK YOU. MOTION CARRIES RECOMMENDATION TO CONSIDER ESTABLISHMENT OF THE DATE PRIVACY AD HOC COMMITTEE AND THE ESTABLISHMENT OF ADDITIONAL AD HOC COMMITTEES AND OR DISSOLUTION OF EXISTING AD HOC COMMITTEES. OKAY. SO FOR THIS ITEM, UM, RYAN, DO YOU WANNA TALK A LITTLE BIT MORE ABOUT THE DATA PRIVACY AD HOC COMMITTEE AND ON IN THE CURRENT INTENDED MEMBERS AND THE FACT THAT WE [5. 23-51628 Recommendation to consider establishment of the Data Privacy Ad Hoc Committee and the establishment of additional Ad Hoc Committees and/or dissolution of existing Ad Hoc Committees.       Suggested Action: Approve recommendation. ] HAVE AN OPENING? YES. THIS IS THE ITEM FOR THAT. THANK YOU. SO THE DATA PRIVACY AD HOC COMMITTEE, UM, THE ROLE THAT WE HAD INITIALLY ENVISIONED FOR THE COMMISSION WAS TO ASSIST WITH THE DEVELOPMENT OF OUR DATA PRIVACY PLAN IMPLEMENTATION COMPONENTS. SO INCLUDING BUT NOT LIMITED TO HELPING DRAFT A DATA PRIVACY ORDINANCE, UH, DRAFT A PRIVACY IMPACT ASSESSMENT TEMPLATE, DRAFT PROPOSED USE POLICIES. UM, ADDITIONAL RESPONSIBILITIES MIGHT INCLUDE CONDUCTING RESEARCH, INVITING PRIVACY, SUBJECT MATTER EXPERTS TO OUR MEETINGS TO WEIGH IN. UH, AND I THINK THIS WOULD BE, UH, THE AD HOC COMMITTEE WOULD BE A RESOURCE TO ASSIST OUR STAFF WITH THE REVIEW AND DEVELOPMENT OF, UH, DRAFT POLICIES AND TEMPLATES. [01:00:01] AND ACTUALLY IT'S TAPPING INTO YOUR EXPERTISE TO HELP US DEVELOP THOSE TEMPLATES AS WELL. UM, AND I THINK I'LL JUST CLOSE BY SAYING I THINK IT'S THE RIGHT TIME FOR THIS COMMITTEE TO BE FORMED. UM, WE HAVE A DATA PRIVACY ANALYST, OMAR MONCAYO, WHO COULDN'T JOIN US TODAY. HE'S OUT SICK. UM, BUT HE'S DEVELOPED A, A KIND OF A ROADMAP FOR WHERE WE'RE GOING WITH THIS WORK AND HAS MAPPED IT OUT TO THE VARIOUS MONTHS OF 2024. SO I THINK IT'S, UM, IT'S AN APPROPRIATE TIME FOR THIS COMMITTEE TO FORM, UH, THE INITIAL COMMISSIONERS WHO HAD VOLUNTEERED INCLUDE, UM, CHAIR BRUNSON, COMMISSIONER DE ARMAS, AND FORMER VICE CHAIR HECS. SO THERE IS AN OPENING. IF WE DO WANNA HAVE A THIRD PERSON AS PART OF THIS GROUP, I WOULD LIKE TO JOIN COMMISSIONER KELLY. IF YOU'RE INTERESTED IN A SPOT, YOU CAN HAVE THE ONE THAT I'M SITTING AND YOU WOULD DRAW. OKAY. YEAH, ABSOLUTELY. YEAH. I'LL TAKE COMMISSIONER DE ARM'S SPOT AND THEN JUST SAY FOR THE FUTURE THAT I AM READY FOR THAT STRATEGIC PLANNING ONE. I'VE BEEN WAITING WHEN IT COMES, I'M GONNA JOIN IT, I PROMISE, BUT I'LL, I'LL JOIN THIS ONE FOR NOW. OKAY, WONDERFUL. SO WE HAVE THE COMMISSION, THE THREE COMMISSIONERS, THEY'LL BE INVOLVED. UM, BUT DID WE, DID WE WANT TO CREATE THE STRATEGIC PLANNING AD HOC COMMITTEE TOO AT THIS TIME? AND DO WE HAVE ANY VOLUNTEERS BESIDES YOU, UM, REMIND US AND I, AND BECAUSE WE DO HAVE A NEW COMMISSIONER, WHAT SOME OF WHAT SOME OF THOSE, UM, GOALS WOULD BE FOR THAT COM COMMITTEE? YEAH, ABSOLUTELY. SO AS I MENTIONED EARLIER, UM, OUR DEPARTMENT IS IN THE PROCESS OF DEVELOPING A SHORT TERM IT STRATEGIC PLAN, WHICH WOULD ARTICULATE, UH, VALUES AND PRINCIPLES AND ALSO AN IMPLEMENTATION ROADMAP OVER THE NEXT TWO YEARS FOR OUR TECHNOLOGY AND INNOVATION DEPARTMENT. I THINK THIS HAS BEEN A PRIORITY OF OUR, UM, DIRECTOR FOR A WHILE. UH, AND SOMETHING THAT WE CAN USE TO MAKE SURE THAT WE'RE PROVIDING MODERN AND ACCESSIBLE SERVICES TO OUR PARTNER DEPARTMENTS AND ALSO TO THE GENERAL PUBLIC. UM, SO SOME OF THE POSSIBLE COMMISSION OR AD HOC COMMITTEE ROLES WOULD BE TO REVIEW AND PROVIDE INPUT ON THE DRAFT STRATEGIC PLAN, UH, SUPPORT OUR, OUR PROPOSED COMMUNITY OUTREACH. UH, ACTUALLY AS PART OF THIS PROCESS, WHICH IS FAIRLY UNIQUE FOR AN IT STRATEGIC PLAN, WE ARE PLANNING TO HOLD COMMUNITY MEETINGS AND FOCUS GROUPS TO GET INPUT FROM LONG BEACH RESIDENTS. SO AS AN EXAMPLE, WE, WE WOULD USE FOR SUPPORTING, PROMOTING THOSE AND FACILITATING THOSE, UM, AND, AND RECRUITING PARTICIPANTS TO JOIN THOSE. AND THEN I THINK A THIRD ROLE COULD BE REVIEWING THE FINAL DRAFT PLAN, BUT I THINK WITH ALL OF THESE AD HOC COMMITTEES, WE WOULD ALSO LOOK FOR YOU TO BRING IN YOUR OWN EXPERTISE FROM YOUR OWN, UH, EXPERIENCES AND RESPONSIBILITIES AND ROLES INTO THIS WORK TOO, TO MAKE SURE THAT OUR STRATEGIC PLAN IS REFLECTING VALUES THAT ARE IMPORTANT TO YOU AS, AS RESIDENTS, BUT ALSO AS TECHNOLOGISTS. MY, UM, ONLY OTHER QUESTION WOULD BE, UM, FOR THE COMMISSIONERS THAT ARE NOT PRESENT, UM, HOW CAN WE ALLOW FOR THEM TO PUT THEIR HAT IN IF, FOR THE STRATEGIC PLANNING ONE? 'CAUSE I KNOW THAT, IF I REMEMBER, I KNOW YOU'VE ALWAYS SAID SO, BUT I THINK THAT THERE WAS SOMEBODY ELSE THAT SAID THEY WANTED TO BE ON THERE. DO WE HAVE THAT NOTED OR THE ONLY COMMISSIONER WHO I HAD NOTED PREVIOUSLY WAS COMMISSIONER KELLY. OH, OKAY. FOR THIS, FOR THIS ITEM. BUT, UM, THERE'S ALWAYS OPPORTUNITIES FOR THE ENTIRE COMMISSION TO BE CONSULTED AS WELL. AND FORUMS LIKE THIS THROUGH PRESENTATIONS AND FEEDBACK, UM, WE CAN ALSO SEND DOCUMENTS, UM, THROUGH ALEJANDRA AND THE INTERIM AND ASK FOR FEEDBACK THAT WAY. AND OF COURSE THE ENTIRE COMMISSION IS WELCOME TO PROMOTE LIKE PLANNED COMMUNITY OUTREACH EVENTS AS WELL. I THINK THE AD HOC GROUP WOULD HAVE MORE OF A, I GUESS, MORE OF AN INTIMATE, UH, ROLE IN SOME OF THOSE DISCUSSIONS AND, AND PLANNING THE PROPOSED OUTREACH THOUGH IN THAT SMALLER GROUP SETTING. SO IT HAS TO BE TWO TWO OR AT LEAST TWO AND NO MORE THAN THREE. THREE. SO ARE THERE TWO MEMBERS THAT WANNA BE, UM, THAT'S, YOU HAVE THREE EAGER PEOPLE HERE. . I, I AM RAISING MY HAND AS WELL FOR, FOR A SEAT ON THAT AD HOC COMMITTEE. RIGHT. WE GOT I LIKEWISE, AND SPEAKS TO ME. WONDERFUL. SO WE HAVE, WE HAVE OUR THREE COMMISSIONERS FOR THIS AS WELL. SO COMMISSIONER DEMA, ARMES COMMISSIONER, WOODMAN NANCE, AND COMMISSIONER KELLY. YEAH. AND I WILL JUST CAVEAT THIS ONE THAT WE'RE PROBABLY STILL A FEW MONTHS OUT FROM CONVENING OUR FIRST AD HOC COMMITTEE MEETING BECAUSE WE'RE ONLY HAVING OUR INTERNAL KICKOFF ABOUT A MONTH FROM NOW. SO JUST WANNA MENTION THAT. SO I THINK WE JUST NEED A MOTION TO ESTABLISH [01:05:01] THE DATA PRIVACY AD HOC COMMITTEE AND THE, UM, STRATEGIC PLANNING, STRATEGIC TECHNOLOGY TECHNOLOGY, STRATEGIC PLAN AD HOC COMMITTEE WITH THE MEMBERS THAT WE DISCUSSED. SO THE MAKER OF THE MOTION, UM, COMMISSIONER DE ARMAS, IF YOU WOULD ACCEPT THAT FRIENDLY THAT THEY DESCRIBED TO ADDING THE SECOND, UM, AD HOC COMMITTEE. SAY, I'M SORRY, HELP ME ACCOUNT AGAIN. SO WOULD YOU, THE STAFF, UM, YEAH. WOULD YOU ACCEPT THE, THE MOTION TO BE AMENDED TO ALSO INCLUDE ESTABLISHMENT OF THE TECHNOLOGY STRATEGIC PLANNING AD HOC COMMITTEE? YES. THERE WE GO. DO YOU ACCEPT THAT THE MOTION TO BE AMENDED TO INCLUDE THE STRATEGIC PLANNING AD HOC COMMITTEE? I SECOND IT. MAYOR BRUNSON, DO WE HAVE ANY PUBLIC COMMENT ON ITEM NUMBER FIVE? SEEING NONE, PLEASE CAST YOUR VOTES. THANK YOU. MOTION CARRIES. AND OUR ITEM NUMBER SIX, RECOMMENDATION TO HOLD AN ELECTION FOR THE POSITION OF VICE CHAIR OF THE TECHNOLOGY AND INNOVATION COMMISSION. SO EARLIER I SAID MAYBE WE DON'T HAVE TO TAKE A VOTE. UH, HOWEVER, I JUST LEARNED YOU MIGHT NOT BE AT OUR NEXT MEETING, AND THAT'S WHY WE NEED VICE CHAIRS, IS SOMEONE TO CONDUCT THE MEETING WHEN THE CHAIR IS NOT HERE. SO I KNOW IT'S NOT IDEAL THAT WE DON'T HAVE THE FULL COMMISSION HERE, BUT I WOULD RECOMMEND, YOU KNOW, APPOINTING, I MEAN APPOINTING A VICE CHAIR. IF, IF WE'RE READY, DO ANY OF US NOMINATE SOMEONE? YEAH, ANYONE CAN NOMINATE. ANYONE ELSE? I WANNA NOMINATE COMMISSIONER DE ARMAS. I MEAN, YOU'RE HERE. I'LL SUCK IT AND YOU'RE SO GREAT. I DON'T KNOW. I I APPRECIATE THAT. I'LL SECOND IT. . UM, I, THAT'S QUITE AN HONOR AND I, I DO ACCEPT THAT NOMINATION. THANK YOU VERY MUCH. WE, BUT WE CAN ALSO TAKE OTHER NOMINATIONS, IS THAT RIGHT? THAT'S CORRECT. ARE THERE ANY OTHER NOMINATIONS OR NO OTHER NOMINATIONS? DO WE HAVE PUBLIC COMMENT ON ITEM NUMBER SIX? SEEING NONE MEMBERS PLEASE CAST YOUR VOTE. MOTION CARRIES. CONGRATULATIONS. VICE CHAIR DE ARMAS. AT THIS TIME WE ASK FOR ANY PUBLIC COMMENT. SEEING NONE, ARE THERE ANY ANNOUNCEMENTS FROM THE COMMISSIONERS? WELL, I GUESS AS I MENTIONED, UM, I, UH, I HAVE KNEE SURGERY THE DAY AFTER AND I DON'T KNOW WHERE I'M GONNA BE AT IN MY HEAD SPACE, SO I'M NOT A NO YET, BUT I'M LEANING THAT WAY. BUT ALSO, I'M EXCITED TO DO GREAT WORK AND WELCOME AGAIN, UH, COMMISSIONER. GREAT. THANK YOU SO MUCH. WE CAN ADJOURN TODAY'S MEETING AT 4:38 PM THANK YOU FOR JOINING. * This transcript was created by voice-to-text technology. The transcript has not been edited for errors or omissions, it is for reference only and is not the official minutes of the meeting.