Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- These notes may have errors and omissions. I couldn't get the names of a lot of the speakers and there are some places where I was thinking or distracted.
- I make no claims as to the completeness of this information
- * Algorithmic transparency legislation hearing 10/16/17
- James Vaca, Chair of NYCC committee on technology
- ** 16-96 2017 Measures of transparency when NYC uses algorithms to impose penalties, police persons
- - Requires publication of source code and querying systems with sample data
- **
- - If left unchecked, algorithms can have negative repercussions
- - Algorithms are a way of encoding assumptions
- - the very data behind them can be biased
- - Despite importance to governamce and their problems, they tend to be hidden from public view
- - Unclear what assumptions they are based upon and what
- - Hence, a lack of transparency
- - What is considered to be most efficient?
- - More dificult for members of the city council to advocate for their citizens if variables are tied up in algorithms
- - When there appears to be inequities or shortages in services, we should find out why
- - Vaca seems to be worked up that his district does not have as much police manpower as other districts
- - The ability to make government accountable is obscured by algorithms and democracy is undermined
- - To ensure that city agencies, when utilizing cutting-edge tools, are accountable
- - First city, first legislative body to undertake the issue in the US
- ** Sunderland -- City enterprise software IT, joined by Craig Kamble mayor's office of data & anaytics
- - City services heavily rely on computer programs
- - Notify NYC app: in-source team
- - Several positions that the city may not hire on their own
- - Not making policy, making apps.
- - 16-96 presents significant operational concerns
- *** security concerns and problems
- - Roadmap for bad actors to exploit and abuse
- - meaningful risk to divulge software
- - Scope is all-encompassing, intentionally targets all programs
- - Releasing proprietary code, releasing old source code
- - Testing is not possible (?)
- - IT departments would have to create a new body of software
- *** Unintended consequences
- - Deliver a deluge of information, most of it unrelated to most interesting city services
- - Users could fabricate data to get the responses they want
- - Code is a small part of decision-making
- - Algorithms supplement, rather than supersede decision making progress
- *** Mayor's office
- - Open data includes 3 recent projects. Reviewing a backlog
- - Motivation to create project library closely aligned with legislation
- - Mayor's office vision aligns somewhat
- - Much legislation about transparency, but decisions are cloaked in opaque algorithms
- - MODA works on specific focus areas: agency projects on priority of the mayor, or legislative mandate
- - MODA's goal is to not own any analytics projects long-term, but for specific projects
- *** Questions
- - RAND formula: always opaque, always used, public does not have a right to know
- - Don't know if it's update in 20 years
- - No comprehensive list of data analytics teams in NYC
- - Why doesn't the mayor know about who's using data and analytics
- - Is there no oversight over which agencies employ advanced data analytics
- - At what level is there an understanding of other agencies' use of data and analytics
- - Never been approached with an agency seeking to implement more transparency
- - Open source: thoroughly vetted
- - Most city systems would divulge system architechture details
- - We don't believe in transparency because we're not doing anything
- - It was a new topic for the Enterprise IT (transparency)
- - HRA employs algorithms to detect benefits fraud, no level of human review
- - Human rights commission: studying decision-making (not algorithms)
- - EIT personally does not know about information rendered without human input
- - How does someone get an apartment in public housing? Strictly by computer, with one appeal
- - What are the bases of those decisions?
- - Feedback loops:
- - Officers stationed in places which have lots of nuisnace calls/arrests which generate nuisance arrests
- - Agencies are not watching their own algorithms
- - Open to creation of commission-based body to oversee the use of algorithms?
- *** My own notes
- - EIT architect seems to be work-shy or worried about mass-reimplimentation of software, renegotiation of contracts
- - Mayor's office analyst seems young & that his boss probably told him what to say, idk
- - If NYC funds are paying for NYC software, well, why isn't that open source/transparent?
- ** Witnesses in testimony
- *** (name unknown)
- - legislation reinforces the core of a responsible, equitable government
- - worried about tech companies entering the public spheres
- - we are outsourcing our government to the unknown
- *** Sheena richardson NYCLU
- - CLU stuff. Much in favor, full testimony submitted
- - (had to answer a couple texts, incomplete notes)
- *** Research fellow at Cornell tech
- - multi-year NSF funding in decision making in algorithmic systems
- - bold proposal, exciting
- - does not reach critical threshold in research
- - privacy implications
- - no guarantees of accuracy or fitness of use
- - any proprietary claims no matter how broad, will fall short of the law
- - black box: administratively burdensome, testing usually takes thousands of queries
- *** Rachel -- Brennan center for justice
- - restoring proper flow of information from government to people
- - filed FOIA to NYPD over predictive policing technologies
- - NYPD expected to spend 45 million on predictive policing software over 5 years
- - 0 records returnd from FOIA, followed suit
- - no disclosure of source code
- - had a hearing in August, and now is before a judge
- - lawsuit filed in July, suit in December, hearing in January
- - striking, the number of exemptions that were brought
- - the NYPD strategy was to wait for a lawsuit to force documents
- *** Alex Kroff
- - no certifications required to create algorithms for city government, but cosmetology licenses required. Seems backwards.
- *** Bronx defenders
- - want to bring to attention of the public a specific algorithm and to make sure that hey are just and fair:
- - NYC developing with private contractor to predict defendent's likelihood in appearing in court
- - would be deployed in bail hearing
- - would increase pre-trial detention in NYC
- - can't predict, but attempts to do so would run aground of bail reform
- - concerned about racial justice aspects of algorithms, exacerbate existing racial disparities
- - one reccommendations: transparency & accountability are good first steps
- - before city algorithms are applied (in courts) the city would be required to perform equity assessments
- *** Brooklyn defenders
- - bail reform & algorithmic transparency, ore algorithms, more policing, more CRJ
- - multinational surety companies have popped up to take advantage of overpolicing
- - RAI's: discriminatory masures like priors, homelessness, education, etc.
- - investigation found accuracy only slightly more accurate than a coin flip
- - RAI's bypass an individual's right to due process
- - underlying data is not transparent, even though engineers say that they are
- *** Security concerns
- - constitutional protections vs security risks: const. protections must take precedence
- - "post-equifax world"
- - people are not consenting to their data being used for overpolicing
- *** Robert Wallace - Research scientist - div of epidemiology
- - algorithms for model systems
- - Rand models that nobody can see & damage data
- - response time is a good index for ambulance
- - for a fire you 'have to build a hospital around apatient'
- - empirical damage measures must be used to determine fire policy
- - you will target tenements for reduction of fire services in high fire neighborhoods
- - models of rand's quality, you wouldn't use for fish! But they are for humans
- - rand hasn't changed since 1970's
- - cuts to city services based on models with questionable validity
- - e.g. city island
- - with global warming, more hurricanes, still using ancient models
- *** Selene - Policy director for Tech NYC
- - nonprofit trade group, increased engagement from tech companies in politics (??)
- - we believe in transparency, treating residents fairly
- - imposing protocols to publish sensitive source code is bad
- - chilling effect on companies publishing code online
- - protecting confidential data
- - (basically against legislation -- so why are you "for transparency" again?)
- *** Josh - Staff attorney for Decarceration, legal aid society
- - brunt of new algorithms have been shoulderd by constituents in
- - may result in wrongful convictions
- - hidden from public view
- - 6 areas where algorithms are used in criminal justice
- - bail, predicitve policing, DNA, family court, parole proceedings, sex offender registration
- - 2 algorithms in bail
- - 17 million dollar prgram with 3000 spots, determined by algorithm
- *** Julie - DNA unit of egal aid society
- - dna evidence & challenges dna certification software in the courtroom
- - fst: probabilitic gene interpreting algorithm
- - no idea how fst calculations are formed, no way to verify soundness of conclusions
- - people went to prison, lost their child from fst
- - fst's source code contained an error
- - hope is that the entire source code
- - strmix - 2 verified errors
- - different algorithms will get different answers in the same case
- - only way to verify that questionable forensic software is not used is to go open source
- *** Center for information technology policy
- - NSF funded research institute
- - bill requires significant changes
- - algo transparency cannot be achieved without data transparency
- - results must be interpretable
- - making source code available is a good 1st step, but must be readable and complete
- - The same algorithm may exhibit two different results with two different training sets
- - Scoring methods for schools and students are different
- - propose following interpretation of transparency:
- - agents must make available details of data collection, summaries of statistical collections of the data
- - privacy-preserving synthetic data
- - 1st mention of EU legislation
- *** Charlie Moffett - NYU graduate student
- - conducted research on behalf of data officers in SF, but also countrywide
- - most research echo what has already been said here today
- - extra recommendations to that committee:
- - regarding publishing source code, often most folks won't understand it
- - understanding outcomes, not just the process
- - algorithms must be clear about confidence and data quality
- - addressing explainability
- - question the use of an algorithm at all, if they can't be explained to the public
- - burden should fall on the vendor
- - reactive auditing: not good
- - leveraging this position when contracting with vendors
- - plan in place for when algorithms go wrong or if mistakes go wrong, what is the redress?
- - transparency must be understandable, and should be inclusive
- *** William Banfield - Tech Worker
- - value of open source
- - 2013 company had incorrect implementation of RAFT protocol. Users downloaded, provided fixes, publicly
- - private company's software could have resulted in data loss
- - regarding security: not a practical way to enforce it
- - openSSL is a public project, is the standard of TLS, government
- - keeping things a secret to improve security is silly.
- *** Sumana - Tech Worker / Recurse Alum
- - Speaking as consultant programmer, citizen
- - 1: tech NYC does not speak for me
- - open source and transparency are the way to better security
- - if there are businesses that make money from citizens data, we need to hold them accountable
- - phrasings of analytics, data, etc require more scrutiny in the context in the law
- - trade secrets, proprietary code: must fix procurement process.
- - code you write with taxpayer money should be made public
- - food report cards: easily understandable
- - if they want to understand deeper, resources are available
- *** Bert - Google, speaking as a citizen
- - objections raised are about existing programs
- - not as many concerns about security models or engineers rehashing old code
- - new development can be held to a high standard of transparency by default
- - centralizing of information and review
- - people of new york should be able to obtain a list of all programs that police persons, penalize, etc
- *** Alex Rich - Data scientist at NYU
- - on bail decisions:
- - open source can be quite understandable
- - new systems can allow people to understand how decisions are made for people
- *** Other thoughts
- - appeal is based on pleading, but you don't have that kind of information
- - people need to know what government decisions are made and on what basis
- - introducing the need for transparency ends up exposing tons of problems, inefficiencies and problems, as a side-efect
Add Comment
Please, Sign In to add comment