The ITx Rutherford 2019 Programme may change without notice
In recent years, there has been a push in the Artificial intelligence (AI) field to simplify the application of machine learning so that its use can be more widely adopted. While this reduces barriers to entry for AI and machine learning, they also introduce the risk that persons or organisations with insufficient expertise will have the ability to use these systems to make decisions that have a significant impact on society based on discriminatory factors. Implementers and decision-makers need to have a good understanding of the features that the system might use and infer from to make predictions, and how these can affect their stakeholders.
In this paper, we outline the risks of this phenomena occurring in a specific case – the application of machine learning applied to secondary school student grades. We demonstrate that naïve approaches can have unanticipated consequences and can generate predictions based on discriminatory factors such as gender or race. The impact of the application of such flawed decisions in matters such as awards, scholarships etc. could entrench detrimental bias in the education systems.
Sunitha Prabhu is a Senior Academic Staff member at the Centre for Information Technology, Waikato Institute of Technology (Wintec).
Sunitha has over 20 years of experience teaching Information Technology in New Zealand. She specialises in teaching Mathematics, Programming, Databases, and supervises IT projects and internships.
Riley is the CEO of TX Labs, a start-up driving innovation and disruption in several industries such as utilities retail and automated expert consultancy. Having completed a Master of Information Technology with a heavy focus on machine learning and artificial intelligence, he currently works as a team lead for his team of software developers to bridge the gap between research and practice alongside his executive role.
With a strong interest in the future of AI with regard to ethics and societal impact, Riley's current research focuses on novel applications of machine learning to existing contexts and the effects of doing so.